is the number assigned to a
character: for example, in EBCDIC the character "A" is usually assigned
the number 193. In Unicode the character "A" is assigned the number 65.
This causes a problem with the semantics of the pack/unpack "U", which
are supposed to pack Unicode code points to characters and back to numbers.
The problem is: which code points to use for code points less than 256?
(for 256 and over there's no problem: Unicode code points are used)
In EBCDIC, for the low 256 the EBCDIC code points are used. This
means that the equivalences
pack("U", ord($character)) eq $character
unpack("U", $character) == ord $character
will hold. (If Unicode code points were applied consistently over
all the possible code points, pack("U",ord("A")) would in EBCDIC
equal I or chr(101), and unpack("U", "A") would equal
65, or I, not 193, or ord "A".)
=head2 Remaining Perl Unicode problems in EBCDIC
=over 4
=item *
Many of the remaining problems seem to be related to case-insensitive matching
=item *
The extensions Unicode::Collate and Unicode::Normalized are not
supported under EBCDIC, likewise for the encoding pragma.
=back
=head2 Unicode and UTF
UTF stands for C.
UTF-8 is an encoding of Unicode into a sequence of 8-bit byte chunks, based on
ASCII and Latin-1.
The length of a sequence required to represent a Unicode code point
depends on the ordinal number of that code point,
with larger numbers requiring more bytes.
UTF-EBCDIC is like UTF-8, but based on EBCDIC.
You may see the term C character or code point.
This simply means that the character has the same numeric
value when encoded as when not.
(Note that this is a very different concept from L
mentioned above.)
For example, the ordinal value of 'A' is 193 in most EBCDIC code pages,
and also is 193 when encoded in UTF-EBCDIC.
All variant code points occupy at least two bytes when encoded.
In UTF-8, the code points corresponding to the lowest 128
ordinal numbers (0 - 127: the ASCII characters) are invariant.
In UTF-EBCDIC, there are 160 invariant characters.
(If you care, the EBCDIC invariants are those characters
which have ASCII equivalents, plus those that correspond to
the C1 controls (80..9f on ASCII platforms).)
A string encoded in UTF-EBCDIC may be longer (but never shorter) than
one encoded in UTF-8.
=head2 Using Encode
Starting from Perl 5.8 you can use the standard new module Encode
to translate from EBCDIC to Latin-1 code points.
Encode knows about more EBCDIC character sets than Perl can currently
be compiled to run on.
use Encode 'from_to';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is in EBCDIC code points
from_to($a, $ebcdic{ord '^'}, 'latin1');
# $a is ISO 8859-1 code points
and from Latin-1 code points to EBCDIC code points
use Encode 'from_to';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is ISO 8859-1 code points
from_to($a, 'latin1', $ebcdic{ord '^'});
# $a is in EBCDIC code points
For doing I/O it is suggested that you use the autotranslating features
of PerlIO, see L.
Since version 5.8 Perl uses the new PerlIO I/O library. This enables
you to use different encodings per IO channel. For example you may use
use Encode;
open($f, ">:encoding(ascii)", "test.ascii");
print $f "Hello World!\n";
open($f, ">:encoding(cp37)", "test.ebcdic");
print $f "Hello World!\n";
open($f, ">:encoding(latin1)", "test.latin1");
print $f "Hello World!\n";
open($f, ">:encoding(utf8)", "test.utf8");
print $f "Hello World!\n";
to get four files containing "Hello World!\n" in ASCII, CP 0037 EBCDIC,
ISO 8859-1 (Latin-1) (in this example identical to ASCII since only ASCII
characters were printed), and
UTF-EBCDIC (in this example identical to normal EBCDIC since only characters
that don't differ between EBCDIC and UTF-EBCDIC were printed). See the
documentation of Encode::PerlIO for details.
As the PerlIO layer uses raw IO (bytes) internally, all this totally
ignores things like the type of your filesystem (ASCII or EBCDIC).
=head1 SINGLE OCTET TABLES
The following tables list the ASCII and Latin 1 ordered sets including
the subsets: C0 controls (0..31), ASCII graphics (32..7e), delete (7f),
C1 controls (80..9f), and Latin-1 (a.k.a. ISO 8859-1) (a0..ff). In the
table non-printing control character names as well as the Latin 1
extensions to ASCII have been labelled with character names roughly
corresponding to I albeit with
substitutions such as s/LATIN// and s/VULGAR// in all cases,
s/CAPITAL LETTER// in some cases, and s/SMALL LETTER ([A-Z])/\l$1/
in some other cases. The "names" of the controls listed here are
the Unicode Version 1 names, except for the few that don't have names, in which
case the names in the Wikipedia article were used
(L).
The differences between the 0037 and 1047 sets are
flagged with ***. The differences between the 1047 and POSIX-BC sets
are flagged with ###. All ord() numbers listed are decimal. If you
would rather see this table listing octal values then run the table
(that is, the pod version of this document since this recipe may not
work with a pod2_other_format translation) through:
=over 4
=item recipe 0
=back
perl -ne 'if(/(.{43})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/)' \
-e '{printf("%s%-9.03o%-9.03o%-9.03o%.03o\n",$1,$2,$3,$4,$5)}' \
perlebcdic.pod
If you want to retain the UTF-x code points then in script form you
might want to write:
=over 4
=item recipe 1
=back
open(FH,") {
if (/(.{43})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/)
{
if ($7 ne '' && $9 ne '') {
printf(
"%s%-9.03o%-9.03o%-9.03o%-9.03o%-3o.%-5o%-3o.%.03o\n",
$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9.03o%-9.03o%-9.03o%-9.03o%-3o.%-5o%.03o\n",
$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9.03o%-9.03o%-9.03o%-9.03o%-9.03o%.03o\n",
$1,$2,$3,$4,$5,$6,$8);
}
}
}
If you would rather see this table listing hexadecimal values then
run the table through:
=over 4
=item recipe 2
=back
perl -ne 'if(/(.{43})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)/)' \
-e '{printf("%s%-9.02X%-9.02X%-9.02X%.02X\n",$1,$2,$3,$4,$5)}' \
perlebcdic.pod
Or, in order to retain the UTF-x code points in hexadecimal:
=over 4
=item recipe 3
=back
open(FH,") {
if (/(.{43})(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\s+(\d+)\.?(\d*)\s+(\d+)\.?(\d*)/)
{
if ($7 ne '' && $9 ne '') {
printf(
"%s%-9.02X%-9.02X%-9.02X%-9.02X%-2X.%-6.02X%02X.%02X\n",
$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9.02X%-9.02X%-9.02X%-9.02X%-2X.%-6.02X%02X\n",
$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9.02X%-9.02X%-9.02X%-9.02X%-9.02X%02X\n",
$1,$2,$3,$4,$5,$6,$8);
}
}
}
ISO 8859-1 CCSID CCSID CCSID 1047
chr CCSID 0819 0037 1047 POSIX-BC UTF-8 UTF-EBCDIC
----------------------------------------------------------------------------------------------
0 0 0 0 0 0
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 55 55 55 4 55
5 45 45 45 5 45
6 46 46 46 6 46
7 47 47 47 7 47
8 22 22 22 8 22
9 5 5 5 9 5
10 37 21 21 10 21 ***
11 11 11 11 11 11