PERLEBCDIC(1) Perl Programmers Reference Guide PERLEBCDIC(1)
NAME
perlebcdic - Considerations for running Perl on EBCDIC platforms
DESCRIPTION
An exploration of some of the issues facing Perl programmers on EBCDIC
based computers. We do not cover localization, internationalization,
or multi byte character set issues other than some discussion of UTF-8
and UTF-EBCDIC.
Portions that are still incomplete are marked with X.
COMON CHARACTER CODE SETS
ASCI
The American Standard Code for Information Interchange is a set of
integers running from 0 to 127 (decimal) that imply character interpre-
tation by the display and other system(s) of computers. The range
0..127 can be covered by setting the bits in a 7-bit binary digit,
hence the set is sometimes referred to as a "7-bit ASCI". ASCI was
described by the American National Standards Institute document ANSI
X3.4-1986. It was also described by ISO 646:1991 (with localization
for currency symbols). The full ASCI set is given in the table below
as the first 128 elements. Languages that can be written adequately
with the characters in ASCI include English, Hawaiian, Indonesian,
Swahili and some Native American languages.
There are many character sets that extend the range of integers from
0..2**7-1 up to 2**8-1, or 8 bit bytes (octets if you prefer). One
common one is the ISO 8859-1 character set.
ISO 8859
The ISO 8859-$n are a collection of character code sets from the Inter-
national Organization for Standardization (ISO) each of which adds
characters to the ASCI set that are typically found in European lan-
guages many of which are based on the Roman, or Latin, alphabet.
Latin 1 (ISO 8859-1)
A particular 8-bit extension to ASCI that includes grave and acute
accented Latin characters. Languages that can employ ISO 8859-1
include all the languages covered by ASCI as well as Afrikaans, Alba-
nian, Basque, Catalan, Danish, Faroese, Finnish, Norwegian, Portuguese,
Spanish, and Swedish. Dutch is covered albeit without the ij ligature.
French is covered too but without the oe ligature. German can use ISO
8859-1 but must do so without German-style quotation marks. This set
is based on Western European extensions to ASCI and is commonly
encountered in world wide web work. In IBM character code set identi-
fication terminology ISO 8859-1 is also known as CSID 819 (or some-
times 0819 or even 00819).
EBCDIC
The Extended Binary Coded Decimal Interchange Code refers to a large
collection of slightly different single and multi byte coded character
sets that are different from ASCI or ISO 8859-1 and typically run on
host computers. The EBCDIC encodings derive from 8 bit byte extensions
of Hollerith punched card encodings. The layout on the cards was such
that high bits were set for the upper and lower case alphabet charac-
ters [a-z] and [A-Z], but there were gaps within each latin alphabet
range.
Some IBM EBCDIC character sets may be known by character code set iden-
tification numbers (CSID numbers) or code page numbers. Leading zero
digits in CSID numbers within this document are insignificant. E.g.
CSID 0037 may be referred to as 37 in places.
13 variant characters
Among IBM EBCDIC character code sets there are 13 characters that are
often mapped to different integer values. Those characters are known
as the 13 "variant" characters and are:
\ [ ] { } ^ ~ ! # $ @ `
000037
Character code set ID 0037 is a mapping of the ASCI plus Latin-1 char-
acters (i.e. ISO 8859-1) to an EBCDIC set. 0037 is used in North Amer-
ican English locales on the OS/400 operating system that runs on AS/400
computers. CSID 37 differs from ISO 8859-1 in 237 places, in other
words they agree on only 19 code point values.
10047
Character code set ID 1047 is also a mapping of the ASCI plus Latin-1
characters (i.e. ISO 8859-1) to an EBCDIC set. 1047 is used under Unix
System Services for OS/390 or z/OS, and OpenEdition for VM/ESA. CSID
1047 differs from CSID 0037 in eight places.
POSIX-BC
The EBCDIC code page in use on Siemens' BS2000 system is distinct from
1047 and 0037. It is identified below as the POSIX-BC set.
Unicode code points versus EBCDIC code points
In Unicode terminology a code point is the number assigned to a charac-
ter: for example, in EBCDIC the character "A" is usually assigned the
number 193. In Unicode the character "A" is assigned the number 65.
This causes a problem with the semantics of the pack/unpack "U", which
are supposed to pack Unicode code points to characters and back to num-
bers. The problem is: which code points to use for code points less
than 256? (for 256 and over there's no problem: Unicode code points
are used) In EBCDIC, for the low 256 the EBCDIC code points are used.
This means that the equivalences
pack("U", ord($character)) eq $character
unpack("U", $character) == ord $character
will hold. (If Unicode code points were applied consistently over all
the possible code points, pack("U",ord("A")) would in EBCDIC equal A
with acute or chr(101), and unpack("U", "A") would equal 65, or non-
breaking space, not 193, or ord "A".)
Remaining Perl Unicode problems in EBCDIC
]o Many of the remaining seem to be related to case-insensitive match-
ing: for example, "/[\x{131}]/" (LATIN SMAL LETER DOTLES I) does
not match "I" case-insensitively, as it should under Unicode. (The
match succeeds in ASCI-derived platforms.)
]o The extensions Unicode::Collate and Unicode::Normalized are not
supported under EBCDIC, likewise for the encoding pragma.
Unicode and UTF
UTF is a Unicode Transformation Format. UTF-8 is a Unicode conforming
representation of the Unicode standard that looks very much like ASCI.
UTF-EBCDIC is an attempt to represent Unicode characters in an EBCDIC
transparent manner.
Using Encode
Starting from Perl 5.8 you can use the standard new module Encode to
translate from EBCDIC to Latin-1 code points
use Encode 'fromto';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is in EBCDIC code points
fromto($a, $ebcdic{ord '^'}, 'latin1');
# $a is ISO 8859-1 code points
and from Latin-1 code points to EBCDIC code points
use Encode 'fromto';
my %ebcdic = ( 176 => 'cp37', 95 => 'cp1047', 106 => 'posix-bc' );
# $a is ISO 8859-1 code points
fromto($a, 'latin1', $ebcdic{ord '^'});
# $a is in EBCDIC code points
For doing I/O it is suggested that you use the autotranslating features
of PerlIO, see perluniintro.
Since version 5.8 Perl uses the new PerlIO I/O library. This enables
you to use different encodings per IO channel. For example you may use
use Encode;
open($f, ">:encoding(ascii)", "test.ascii");
print $f "Hello World!\n";
open($f, ">:encoding(cp37)", "test.ebcdic");
print $f "Hello World!\n";
open($f, ">:encoding(latin1)", "test.latin1");
print $f "Hello World!\n";
open($f, ">:encoding(utf8)", "test.utf8");
print $f "Hello World!\n";
to get two files containing "Hello World!\n" in ASCI, CP 37 EBCDIC,
ISO 8859-1 (Latin-1) (in this example identical to ASCI) respective
UTF-EBCDIC (in this example identical to normal EBCDIC). See the docu-
mentation of Encode::PerlIO for details.
As the PerlIO layer uses raw IO (bytes) internally, all this totally
ignores things like the type of your filesystem (ASCI or EBCDIC).
SINGLE OCTET TABLES
The following tables list the ASCI and Latin 1 ordered sets including
the subsets: C0 controls (0..31), ASCI graphics (32..7e), delete (7f),
C1 controls (80..9f), and Latin-1 (a.k.a. ISO 8859-1) (a0..ff). In the
table non-printing control character names as well as the Latin 1
extensions to ASCI have been labelled with character names roughly
corresponding to The Unicode Standard, Version 3.0 albeit with substi-
tutions such as s/LATIN/ and s/VULGAR/ in all cases, s/CAPITAL LET-
TER/ in some cases, and s/SMAL LETER ([A-Z])/\l$1/ in some other
cases (the "charnames" pragma names unfortunately do not list explicit
names for the C0 or C1 control characters). The "names" of the C1 con-
trol set (128..159 in ISO 8859-1) listed here are somewhat arbitrary.
The differences between the 0037 and 1047 sets are flagged with ***.
The differences between the 1047 and POSIX-BC sets are flagged with
###. All ord() numbers listed are decimal. If you would rather see
this table listing octal values then run the table (that is, the pod
version of this document since this recipe may not work with a
pod2otherformat translation) through:
recipe 0
perl -ne 'if(/(.{33})(\d])\s](\d])\s](\d])\s](\d])/)' \
-e '{printf("%s%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5)}' perlebcdic.pod
If you want to retain the UTF-x code points then in script form you
might want to write:
recipe 1
open(FH,") {
if (/(.{33})(\d])\s](\d])\s](\d])\s](\d])\s](\d])\.?(\d*)\s](\d])\.?(\d*)/) {
if ($7 ne '' && $9 ne '') {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%-3o.%o\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9o%-9o%-9o%-9o%-3o.%-5o%o\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9o%-9o%-9o%-9o%-9o%o\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
If you would rather see this table listing hexadecimal values then run
the table through:
recipe 2
perl -ne 'if(/(.{33})(\d])\s](\d])\s](\d])\s](\d])/)' \
-e '{printf("%s%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5)}' perlebcdic.pod
Or, in order to retain the UTF-x code points in hexadecimal:
recipe 3
open(FH,") {
if (/(.{33})(\d])\s](\d])\s](\d])\s](\d])\s](\d])\.?(\d*)\s](\d])\.?(\d*)/) {
if ($7 ne '' && $9 ne '') {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%-2X.%X\n",$1,$2,$3,$4,$5,$6,$7,$8,$9);
}
elsif ($7 ne '') {
printf("%s%-9X%-9X%-9X%-9X%-2X.%-6X%X\n",$1,$2,$3,$4,$5,$6,$7,$8);
}
else {
printf("%s%-9X%-9X%-9X%-9X%-9X%X\n",$1,$2,$3,$4,$5,$6,$8);
}
}
}
incomp- incomp-
8859-1 lete lete
chr 0819 0037 1047 POSIX-BC UTF-8 UTF-EBCDIC
------------------------------------------------------------------------------------
0 0 0 0 0 0
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 55 55 55 4 55
5 45 45 45 5 45
6 46 46 46 6 46
7 47 47 47 7 47
8 22 22 22 8 22
9 5 5 5 9 5
10 37 21 21 10 21 ***
11 11 11 11 11 11
|
