You In Here :
/
/
proc /
self /
root /
usr /
share /
perl5 /
pod /=head1 NAME
perlunicode - Unicode support in Perl
=head1 DESCRIPTION
=head2 Important Caveats
Unicode support is an extensive requirement. While Perl does not
implement the Unicode standard or the accompanying technical reports
from cover to cover, Perl does support many Unicode features.
People who want to learn to use Unicode in Perl, should probably read
the L
and
L, before reading
this reference document.
Also, the use of Unicode may present security issues that aren't obvious.
Read L.
=over 4
=item Safest if you "use feature 'unicode_strings'"
In order to preserve backward compatibility, Perl does not turn
on full internal Unicode support unless the pragma
C is specified. (This is automatically
selected if you use C or higher.) Failure to do this can
trigger unexpected surprises. See L below.
This pragma doesn't affect I/O, and there are still several places
where Unicode isn't fully supported, such as in filenames.
=item Input and Output Layers
Perl knows when a filehandle uses Perl's internal Unicode encodings
(UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with
the ":encoding(utf8)" layer. Other encodings can be converted to Perl's
encoding on input or from Perl's encoding on output by use of the
":encoding(...)" layer. See L.
To indicate that Perl source itself is in UTF-8, use C.
=item C still needed to enable UTF-8/UTF-EBCDIC in scripts
As a compatibility measure, the C pragma must be explicitly
included to enable recognition of UTF-8 in the Perl scripts themselves
(in string or regular expression literals, or in identifier names) on
ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based
machines. B
is needed.> See L.
=item BOM-marked scripts and UTF-16 scripts autodetected
If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE,
or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either
endianness, Perl will correctly read in the script as Unicode.
(BOMless UTF-8 cannot be effectively recognized or differentiated from
ISO 8859-1 or other eight-bit encodings.)
=item C needed to upgrade non-Latin-1 byte strings
By default, there is a fundamental asymmetry in Perl's Unicode model:
implicit upgrading from byte strings to Unicode strings assumes that
they were encoded in I, but Unicode strings are
downgraded with UTF-8 encoding. This happens because the first 256
codepoints in Unicode happens to agree with Latin-1.
See L for more details.
=back
=head2 Byte and Character Semantics
Beginning with version 5.6, Perl uses logically-wide characters to
represent strings internally.
Starting in Perl 5.14, Perl-level operations work with
characters rather than bytes within the scope of a
C> (or equivalently
C or higher). (This is not true if bytes have been
explicitly requested by C>, nor necessarily true
for interactions with the platform's operating system.)
For earlier Perls, and when C is not in effect, Perl
provides a fairly safe environment that can handle both types of
semantics in programs. For operations where Perl can unambiguously
decide that the input data are characters, Perl switches to character
semantics. For operations where this determination cannot be made
without additional information from the user, Perl decides in favor of
compatibility and chooses to use byte semantics.
When C (but not C) is in
effect, Perl uses the semantics associated with the current locale.
(C overrides C in the same scope;
while C effectively also selects
C in its scope; see L.)
Otherwise, Perl uses the platform's native
byte semantics for characters whose code points are less than 256, and
Unicode semantics for those greater than 255. On EBCDIC platforms, this
is almost seamless, as the EBCDIC code pages that Perl handles are
equivalent to Unicode's first 256 code points. (The exception is that
EBCDIC regular expression case-insensitive matching rules are not as
as robust as Unicode's.) But on ASCII platforms, Perl uses US-ASCII
(or Basic Latin in Unicode terminology) byte semantics, meaning that characters
whose ordinal numbers are in the range 128 - 255 are undefined except for their
ordinal numbers. This means that none have case (upper and lower), nor are any
a member of character classes, like C<[:alpha:]> or C<\w>. (But all do belong
to the C<\W> class or the Perl regular expression extension C<[:^alpha:]>.)
This behavior preserves compatibility with earlier versions of Perl,
which allowed byte semantics in Perl operations only if
none of the program's inputs were marked as being a source of Unicode
character data. Such data may come from filehandles, from calls to
external programs, from information provided by the system (such as %ENV),
or from literals and constants in the source text.
The C pragma is primarily a compatibility device that enables
recognition of UTF-(8|EBCDIC) in literals encountered by the parser.
Note that this pragma is only required while Perl defaults to byte
semantics; when character semantics become the default, this pragma
may become a no-op. See L.
If strings operating under byte semantics and strings with Unicode
character data are concatenated, the new string will have
character semantics. This can cause surprises: See L, below.
You can choose to be warned when this happens. See L.
Under character semantics, many operations that formerly operated on
bytes now operate on characters. A character in Perl is
logically just a number ranging from 0 to 2**31 or so. Larger
characters may encode into longer sequences of bytes internally, but
this internal detail is mostly hidden for Perl code.
See L for more.
=head2 Effects of Character Semantics
Character semantics have the following effects:
=over 4
=item *
Strings--including hash keys--and regular expression patterns may
contain characters that have an ordinal value larger than 255.
If you use a Unicode editor to edit your program, Unicode characters may
occur directly within the literal strings in UTF-8 encoding, or UTF-16.
(The former requires a BOM or C, the latter requires a BOM.)
Unicode characters can also be added to a string by using the C<\N{U+...}>
notation. The Unicode code for the desired character, in hexadecimal,
should be placed in the braces, after the C. For instance, a smiley face is
C<\N{U+263A}>.
Alternatively, you can use the C<\x{...}> notation for characters 0x100 and
above. For characters below 0x100 you may get byte semantics instead of
character semantics; see L. On EBCDIC machines there is
the additional problem that the value for such characters gives the EBCDIC
character rather than the Unicode one, thus it is more portable to use
C<\N{U+...}> instead.
Additionally, you can use the C<\N{...}> notation and put the official
Unicode character name within the braces, such as
C<\N{WHITE SMILING FACE}>. This automatically loads the L
module with the C<:full> and C<:short> options. If you prefer different
options for this module, you can instead, before the C<\N{...}>,
explicitly load it with your desired options; for example,
use charnames ':loose';
=item *
If an appropriate L is specified, identifiers within the
Perl script may contain Unicode alphanumeric characters, including
ideographs. Perl does not currently attempt to canonicalize variable
names.
=item *
Regular expressions match characters instead of bytes. "." matches
a character instead of a byte.
=item *
Bracketed character classes in regular expressions match characters instead of
bytes and match against the character properties specified in the
Unicode properties database. C<\w> can be used to match a Japanese
ideograph, for instance.
=item *
Named Unicode properties, scripts, and block ranges may be used (like bracketed
character classes) by using the C<\p{}> "matches property" construct and
the C<\P{}> negation, "doesn't match property".
See L for more details.
You can define your own character properties and use them
in the regular expression with the C<\p{}> or C<\P{}> construct.
See L for more details.
=item *
The special pattern C<\X> matches a logical character, an "extended grapheme
cluster" in Standardese. In Unicode what appears to the user to be a single
character, for example an accented C, may in fact be composed of a sequence
of characters, in this case a C followed by an accent character. C<\X>
will match the entire sequence.
=item *
The C functionality has been removed. For similar
functionality see pack('U0', ...) and pack('C0', ...).
=item *
Case translation operators use the Unicode case translation tables
when character input is provided. Note that C, or C<\U> in
interpolated strings, translates to uppercase, while C,
or C<\u> in interpolated strings, translates to titlecase in languages
that make the distinction (which is equivalent to uppercase in languages
without the distinction).
=item *
Most operators that deal with positions or lengths in a string will
automatically switch to using character positions, including
C, C, C, C, C, C,
C, C, and C. An operator that
specifically does not switch is C. Operators that really don't
care include operators that treat strings as a bucket of bits such as
C, and operators dealing with filenames.
=item *
The C/C letter C does I change, since it is often
used for byte-oriented formats. Again, think C in the C language.
There is a new C specifier that converts between Unicode characters
and code points. There is also a C specifier that is the equivalent of
C/C and properly handles character values even if they are above 255.
=item *
The C and C functions work on characters, similar to
C and C, I C and
C. C and C are methods for
emulating byte-oriented C and C on Unicode strings.
While these methods reveal the internal encoding of Unicode strings,
that is not something one normally needs to care about at all.
=item *
The bit string operators, C<& | ^ ~>, can operate on character data.
However, for backward compatibility, such as when using bit string
operations when characters are all less than 256 in ordinal value, one
should not use C<~> (the bit complement) with characters of both
values less than 256 and values greater than 256. Most importantly,
DeMorgan's laws (C<~($x|$y) eq ~$x&~$y> and C<~($x&$y) eq ~$x|~$y>)
will not hold. The reason for this mathematical I is that
the complement cannot return B the 8-bit (byte-wide) bit
complement B the full character-wide bit complement.
=item *
There is a CPAN module, L, which allows you to define
your own mappings to be used in C, C, C,
C, and C (or their double-quoted string inlined
versions such as C<\U>).
(Prior to Perl 5.16, this functionality was partially provided
in the Perl core, but suffered from a number of insurmountable
drawbacks, so the CPAN module was written instead.)
=back
=over 4
=item *
And finally, C reverses by character rather than by byte.
=back
=head2 Unicode Character Properties
(The only time that Perl considers a sequence of individual code
points as a single logical character is in the C<\X> construct, already
mentioned above. Therefore "character" in this discussion means a single
Unicode code point.)
Very nearly all Unicode character properties are accessible through
regular expressions by using the C<\p{}> "matches property" construct
and the C<\P{}> "doesn't match property" for its negation.
For instance, C<\p{Uppercase}> matches any single character with the Unicode
"Uppercase" property, while C<\p{L}> matches any character with a
General_Category of "L" (letter) property. Brackets are not
required for single letter property names, so C<\p{L}> is equivalent to C<\pL>.
More formally, C<\p{Uppercase}> matches any single character whose Unicode
Uppercase property value is True, and C<\P{Uppercase}> matches any character
whose Uppercase property value is False, and they could have been written as
C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively.
This formality is needed when properties are not binary; that is, if they can
take on more values than just True and False. For example, the Bidi_Class (see
L below), can take on several different
values, such as Left, Right, Whitespace, and others. To match these, one needs
to specify both the property name (Bidi_Class), AND the value being
matched against
(Left, Right, etc.). This is done, as in the examples above, by having the
two components separated by an equal sign (or interchangeably, a colon), like
C<\p{Bidi_Class: Left}>.
All Unicode-defined character properties may be written in these compound forms
of C<\p{property=value}> or C<\p{property:value}>, but Perl provides some
additional properties that are written only in the single form, as well as
single-form short-cuts for all binary properties and certain others described
below, in which you may omit the property name and the equals or colon
separator.
Most Unicode character properties have at least two synonyms (or aliases if you
prefer): a short one that is easier to type and a longer one that is more
descriptive and hence easier to understand. Thus the "L" and "Letter" properties
above are equivalent and can be used interchangeably. Likewise,
"Upper" is a synonym for "Uppercase", and we could have written
C<\p{Uppercase}> equivalently as C<\p{Upper}>. Also, there are typically
various synonyms for the values the property can be. For binary properties,
"True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F",
"No", and "N". But be careful. A short form of a value for one property may
not mean the same thing as the same short form for another. Thus, for the
General_Category property, "L" means "Letter", but for the Bidi_Class property,
"L" means "Left". A complete list of properties and synonyms is in
L.
Upper/lower case differences in property names and values are irrelevant;
thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>.
Similarly, you can add or subtract underscores anywhere in the middle of a
word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space
is irrelevant adjacent to non-word characters, such as the braces and the equals
or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are
equivalent to these as well. In fact, white space and even
hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is
equivalent. All this is called "loose-matching" by Unicode. The few places
where stricter matching is used is in the middle of numbers, and in the Perl
extension properties that begin or end with an underscore. Stricter matching
cares about white space (except adjacent to non-word characters),
hyphens, and non-interior underscores.
You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret
(^) between the first brace and the property name: C<\p{^Tamil}> is
equal to C<\P{Tamil}>.
Almost all properties are immune to case-insensitive matching. That is,
adding a C regular expression modifier does not change what they
match. There are two sets that are affected.
The first set is
C,
C,
and C,
all of which match C under C matching.
And the second set is
C,
C,
and C,
all of which match C under C matching.
This set also includes its subsets C and C both
of which under C matching match C.
(The difference between these sets is that some things, such as Roman
numerals, come in both upper and lower case so they are C, but aren't considered
letters, so they aren't Cs.)
The result is undefined if you try to match a non-Unicode code point
(that is, one above 0x10FFFF) against a Unicode property. Currently, a
warning is raised, and the match will fail. In some cases, this is
counterintuitive, as both these fail:
chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails.
chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails!
=head3 B
Every Unicode character is assigned a general category, which is the "most
usual categorization of a character" (from
L).
The compound way of writing these is like C<\p{General_Category=Number}>
(short, C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up
through the equal or colon separator is omitted. So you can instead just write
C<\pN>.
Here are the short and long forms of the General Category properties:
Short Long
L Letter
LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}])
Lu Uppercase_Letter
Ll Lowercase_Letter
Lt Titlecase_Letter
Lm Modifier_Letter
Lo Other_Letter
M Mark
Mn Nonspacing_Mark
Mc Spacing_Mark
Me Enclosing_Mark
N Number
Nd Decimal_Number (also Digit)
Nl Letter_Number
No Other_Number
P Punctuation (also Punct)
Pc Connector_Punctuation
Pd Dash_Punctuation
Ps Open_Punctuation
Pe Close_Punctuation
Pi Initial_Punctuation
(may behave like Ps or Pe depending on usage)
Pf Final_Punctuation
(may behave like Ps or Pe depending on usage)
Po Other_Punctuation
S Symbol
Sm Math_Symbol
Sc Currency_Symbol
Sk Modifier_Symbol
So Other_Symbol
Z Separator
Zs Space_Separator
Zl Line_Separator
Zp Paragraph_Separator
C Other
Cc Control (also Cntrl)
Cf Format
Cs Surrogate
Co Private_Use
Cn Unassigned
Single-letter properties match all characters in any of the
two-letter sub-properties starting with the same letter.
C and C are special: both are aliases for the set consisting of everything matched by C, C, and C.
=head3 B
Because scripts differ in their directionality (Hebrew and Arabic are
written right to left, for example) Unicode supplies these properties in
the Bidi_Class class:
Property Meaning
L Left-to-Right
LRE Left-to-Right Embedding
LRO Left-to-Right Override
R Right-to-Left
AL Arabic Letter
RLE Right-to-Left Embedding
RLO Right-to-Left Override
PDF Pop Directional Format
EN European Number
ES European Separator
ET European Terminator
AN Arabic Number
CS Common Separator
NSM Non-Spacing Mark
BN Boundary Neutral
B Paragraph Separator
S Segment Separator
WS Whitespace
ON Other Neutrals
This property is always written in the compound form.
For example, C<\p{Bidi_Class:R}> matches characters that are normally
written right to left.
=head3 B
The world's languages are written in many different scripts. This sentence
(unless you're reading it in translation) is written in Latin, while Russian is
written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in
Hiragana or Katakana. There are many more.
The Unicode Script and Script_Extensions properties give what script a
given character is in. Either property can be specified with the
compound form like
C<\p{Script=Hebrew}> (short: C<\p{sc=hebr}>), or
C<\p{Script_Extensions=Javanese}> (short: C<\p{scx=java}>).
In addition, Perl furnishes shortcuts for all
C