Globalize your On Demand Business

• Coding of Graphic Character Data
• Elements of Character Data Representation
• Graphic Character Global Identifier
• Long-Form Identification
• Encoding Scheme Identifier
• Coded Graphic Character Set Global Identifier
• Special-Purpose Values for GCSGID and CPGID
• Additional Coding-Related Required Information
• Short-Form Identification
• Coded Character Set Identifier
• Representation of CDRA Identifiers
• CCSID Values
• Tagging in CDRA
• Meaning of Tagging in CDRA

The identifiers associated with graphic character representation (see Figure 7) are:

• Graphic Character Global Identifier and Graphic Character UCS Identifier
• Long-form identification -- a set of identifiers consisting of:
• Encoding Scheme Identifier
• One or more Coded Graphic Character Set Global Identifiers, each of which is a concatenation of:
• A Graphic Character Set Global Identifier
• A Code Page Global Identifier.
• Additional Coding-related Required Information, as specified by the Encoding Scheme Identifier (for example, a list of valid first bytes of double-byte code points, code points used for code extensions, or a set of floating accents and a valid associated character).
• Short-form identification -- an identifier called Coded Character Set Identifier (CCSID) is defined as an alternative to the variable-length long form.

These identifiers are detailed in the following sections.

Figure 7. CDRA Identifier Forms

The GCGID and GCUID are fully described on the IBM Graphic Character Identifiers web page.

In CDRA, the terms graphic character identifier, character identifier, meaning of graphic character, and rendering of graphic character are synonymous with GCGID.

IBM's GCGID system provides for distinguishing between two renderings of a graphic character. When no specific rendering is indicated, a "nominal" rendering is assumed with the character. Specific renderings can further be specified using another identifier such as Font Global Identifier, FGID.

The rendering part of a GCGID is of significance primarily for presentation processing such as formatting, displaying, or printing. GCGIDs with different renderings typically appear in character sets and code pages that are primarily presentation-oriented. However, some of these character sets and code pages are used for all aspects of processing of graphic characters. If they are encountered by functions such as comparison, depending on the context of use, graphic characters with two different renderings may be equated.

When graphic characters with two different renderings are part of a character set and are included in the same coded character set, different code points are assigned different GCGIDs to represent the different renderings. Some examples of use of GCGIDs for graphic characters with different renderings are:

• The different shapes of Arabic characters coded from CS 00235 in CP 00420 or CP 00864
• The wide Latin alphabets (such as A through Z) of Far East double-byte code pages (such as CP 00300), to distinguish them from "nominal" width A through Z in single-byte code pages (such as CP 000290). CP 00290 and CP 00300 are used together in mixed single-byte and double-byte codes (such as CCSID 05026).

The Encoding Scheme Identifier, ESID, is a 4-digit hexadecimal number that identifies the scheme used to code graphic character data. The following 3 elements have been used where possible in ESID definitions.

The basic encoding structure (x)

This element identifies the basic structural characteristic that differentiates various encoding schemes such as EBCDIC, ISO-8, IBM-PC Data, or others.

The number of bytes per code point (y)

When the encoding scheme permits a different number of 7-bit or 8-bit bytes per code point, this element identifies the selection used.

The code extension method (zz)

Code extensions are techniques used to encode more characters than can be accommodated in the basic encoding structure. An example is the use of SO (Shift-Out) and SI (Shift-In) as controls to access an alternative assignment of graphic characters to code points, and to show whether one byte or two bytes of the data constitute a code point, in the EBCDIC mixed single-byte and double-byte encoding.

This element of the ESID identifies the particular method of code extension used from among the many that may be allowed in the encoding scheme.

Note to developers: While efforts have been made to define ESIDs using these elements, not all ESIDs follow the above pattern. It is esential that all encoding scheme identifiers be defined by the owner of CDRA prior to being used.

Figure 8 shows the three components of the ESID. The component values and their meanings are detailed in the following sections.

Figure 8. Encoding Scheme Identifier Format

Figure 8. Encoding Scheme Identifier Format

The ESID makes the following possible:

• The selection of the correct algorithms (such as parsing) to be invoked to process graphic character data.
• Identification of reserved code point(s) for allocation to some most-frequently used characters such as SPACE (GCGID SP010000).

The ESID also determines the number and types of other CDRA identifiers needed in the long form.

The term Encoding Scheme (ES) is synonymous with ESID.

Basic Encoding Structure (x)

The following values are defined for the first nibble (10) of the ESID to identify the structure. The properties of each structure are detailed in Appendix A. Encoding Schemes.

Hex Structure
0 Defaults to higher level in hierarchy
1 EBCDIC
2 IBM-PC Data
3 IBM-PC Display
4 ISO 8
5 ISO 7
6 EBCDIC presentation
7 UCS
8 UCS Display
9 8 bit, for a standalone, 7-bit EUC G-set that has been shifted into the right half of the encoding space
A-C Reserved for future allocation by CDRA
D Unique encoding. Details of the encoding structure are found in the related CP and CCSID definitions.
E Reserved for extending ES id, when needed
F For Private Use. Use of this value must be accompanied by a specification of the structure, and the rules for usage with specific values of the other parts of the ESID. Definition of the Private Use values is outside the scope of CDRA.

Number of Bytes Indicator (y)

An encoding scheme may permit specific variations in the number of bytes associated with a code point (for example, EBCDIC single-byte versus EBCDIC double-byte). These variations are shown using the second nibble of the ESID. The value of this nibble is not the number of bytes per code point, but rather a pointer to the definition. The value does not equate to the number of bytes in the code point. The values representing a variable number of bytes identify what is allowed to appear in a string, not what actually appears. The encoding scheme defines permitted values of this nibble for the encoding structure used.

If the value of the first nibble defining the basic encoding structure element is zero, the second nibble identifying the number of bytes must also be zero.

The following values are defined:

Hex Number of Bytes per Code Point
0 Reserved for use with zero value for the basic encoding structure
1 Fixed single-byte, SBCS
2 Fixed double-byte, DBCS (including ISO/IEC 10646-1 UCS-2)
3 IBM Far East style, mixed single-byte and double-byte
4 ISO 2022 schemes (EUC, TCP/IP)
5 UCS-4 or UTF-32
6 Reserved for future allocation by CDRA
7 Fixed triple-byte
8 UTF-n variable number of bytes, self describing (37)
9 Fixed 4-byte
A Mixed 1-byte, 2-byte, 4-byte (for GB 18030)
B BOCU-1, SCSU and similar Stateful Compression Schemes
C-E Reserved for future allocation by CDRA
F For Private Use. The specification of Private Use must include the values (and the specific meaning) of the encoding structure nibble with which it can be used. Definition of the Private Use values is outside the scope of CDRA.

Code Extension Method (zz)

The code extension method is described by the second byte of the ES identifier. This byte indicates that a code point from an extended coded character set may appear in the data; it does not mean that the extension method has actually been used in a specific character string.

When the first two nibbles of the ESID are zeros, the code extension byte value must be zero.

The following values are defined:

Hex Code Extension Method
00 No extensions are specified
01 Locking Shifts (SO and SI, or LS1 and LS0 (11) or UC and LC locking controls)
02 Reserved for future allocation by CDRA
03 IBM EUC scheme (ISO-2022-based)
04 TCP/IP scheme (ISO-2022-based)
05 ISO-8 with possible graphics in C1 area (X'80' to X'9F')
06 Reserved for future allocation by CDRA
07 UTF-8 Universal Transformation Format
08 UTF-EBCDIC Universal Transformation Format
09 Used for an individual Unicode plane
0A Reserved for future allocation by CDRA
0B Used to indicate Little Endian Order for UCS
0C Unicode Standard Code Compression Scheme
0D Compatibility Encoding Scheme for UTF-16: 8-Bit (CESU-8)
0E Binary Ordered Compression for Unicode (BOCU-1)
0F UCS with Byte Order Mark (BOM) to incidate Endianness; BE is assumed in absence of BOM
10 to 49 Reserved for future allocation by CDRA
50 ISO-7 with possible graphics in the C0 area (X'00'to X'1F')
51 to 54 Reserved for future allocation by CDRA
55 ISO-8 with possible graphics in C0 and C1 areas (X'00' to X'1F' and X'80' to X'9F')
56 to FD Reserved for future allocation by CDRA
FE Reserved for Private Use of Code Extension. Definition of the Private Use value is outside the scope of CDRA.
FF Code Extension consideration does not apply.

Code Extension States

When an encoding scheme uses an extension technique, it uses more than one elementary coded character set to create a composite coded character set. The scheme specifies one code extension switching state for each coded character set used. While in a given state, the associated coded character set is used for representing and interpreting the character data. The method for switching between these states can be implicit or explicit, locking or single shifting. The number of switching states and the method of switching between the states in a coded character set are specified by the encoding scheme. State numbering begins at 1 and increases by 1 for each coded character set. For example, in mixed single-byte, double-byte encodings there are 2 states; the single-byte coded character set is state 1 and the double-byte coded character set is state 2. Encoding schemes which define a single coded character set have a single state; state 1.

The second nibble and the last byte of the ESID together identify the number of switching states. The last byte of the ESID identifies the switching method employed in an encoding scheme. The first and second nibbles identify the nature of the elementary code structures used in the resulting composite structure.

ESID Values

ESID values and their semantics are listed in Figure 9.

Figure 9. ESID values

Coded Graphic Character Set Global Identifier

A GCSGID is assigned to every Registered Graphic Character Set by IBM (or by a customer organization). An explanation of CGCSGIDs is provided in Appendix E. Graphic Character Global Identifiers.

See Special-Purpose Values for GCSGID and CPGID for use of 00000 (X'0000'), 65,535 (X'FFFF') and other special-purpose values for GCSGID.

The term Character Set (CS) is synonymous with GCSGID.

SPACE as a special character

By itself, the GCSGID does not specify either the inclusion or the exclusion of the SPACE (GCGID = SP010000) character. Each encoding scheme reserves one or more code points for allocation to the SPACE character. There are two possible code points for it when using mixed SBCS and DBCS encoding schemes.

Code Page Global Identifier

A Code Page Global Identifier, CPGID, is a 5-digit decimal number assigned to a code page.

A code page is a specification of code points from a defined encoding structure for each graphic character (12) in a collection of one or more graphic character sets.

A CPGID identifies a unique assignment of the graphic code points in an encoding scheme to a specific set of GCGIDs. Many character sets may be contained in a code page. When all of the code points in the graphic encoding space of a code page have been assigned, then the character set containing this collection of GCGIDs is defined to be full. Often, when a code page is first created and registered, some of the assignable graphic code points may not have assigned GCGIDs. The character set containing these assigned characters is defined to be maximal. As more code point assignments are made, the maximal character set will change. Once all code points have been assigned, the maximal set will be the full set.

A CPGID is assigned to every Registered Code Page by IBM. In some cases, the same CPGIDs have been used when the encoding structures are similar. (13) An explanation of CPGIDs is provided in Appendix E. Graphic Character Global Identifiers.

The range of CPGID values is 00001 (X'0001') to 65534 (X'FFFE'). The values X'FE00' to X'FEFF' are reserved for Request for Price Quotation (RPQ) use by IBM products. The values X'FF00' to X'FFFE' are reserved for customer use.

The term Code Page (CP) is synonymous with CPGID.

IBM standards reserve the values X'0000' and X'FFFF' for future assignments. In practice, these identifier values have been used for a number of different special purposes. Some values other than X'0000' and X'FFFF' that have been reserved for special-purpose use are also included in this section. In the interest of providing consistency between various implementations, the semantics of use of these values, either in current use or for future use, are defined here.

Some known definitions are listed below, along with their semantics. Others will be added as they become known to CDRA.

The CS value of X'0000' is used in several IBM architectures, such as Formatted Data Object Content Architecture (FD:OCA), Mixed Object Document Content Architecture (MO:DCA), and Document Interchange Architecture (DIA) Profiles, to facilitate migration and coexistence between the use of only a CGCSGID (CS, CP pair) prior to the advent of CDRA and the use of the CCSID identifier in different architecture definitions.

In these architectures, if the CS portion of a structured field carrying a CGCSGID has a value of X'0000', the value of the CP portion is interpreted as a CCSID. The following definitions then apply:

1. CS X'0000' with CP X'0000'
   The CP value of X'0000' is interpreted as CCSID X'0000'. This CCSID value means that the tag value is to be inherited from a higher level in a hierarchical structure.


2. CS X'0000' with CP X'FFFE'
   The CP value of X'FFFE' is interpreted as CCSID X'FFFE'. This CCSID value means that the tag value is to be obtained from a lower level in a hierarchical structure.


3. CS X'0000' with CP X'FFFF'
   The CP value of X'FFFF' is interpreted as CCSID X'FFFF'. This CCSID value means that the tagged data is to be interpreted as "not graphic character data" or "actual representation is unknown".


4. CS X'0000' with all other CP values
   The CP value is interpreted as a CCSID.

The CS value of X'FFFF' can have the following special-purpose definitions.

Special CS and CP values are used to indicate "No CS, No CP" in the ACRI-EUC structure defined in "ACRI Type EUC (ACRI-EUC)". the following special-purpose value is defined:

1. CS X'FDFF'
   The CS value of X'FDFF' implies that there is no character set, that is that the corresponding G set is not used for this particular EUC CCSID.


2. CP X'FDFF'
   The CP value of X'FDFF' implies that there is no code page, that is that the corresponding G set is not used for this particular EUC CCSID.

Within CDRA the following CS/CP pair have been used in the definition of Unicode CCSIDs.

1. CS X’FFF0’ (65520) and CP X’FFF0’ (65520)
   This CS/CP pair is used to represent an empty plane of Unicode. By definition CS 65520 is an empty set containing no characters and CP 65520 is a Unicode plane with no characters defined.

Additional Coding-Related Required Information

Some encoding schemes require specifications beyond the CS and CP elements to complete their definitions. Such specifications are called Additional Coding-related Required Information (ACRI) elements.

Three types of ACRI are defined below.

ACRI PC Mixed Byte (ACRI-PCMB)

This type of ACRI applies to ES values X'2300', X'2305' or X'3300' (see semantics of these ES values in Figure 9). It cannot be specified with any other ES values. It consists of the specification of ranges of valid first bytes of double-bytes associated with particular CS, CP pairs that are used with this encoding scheme. An ACRI-PCMB has the following format:

Additional Coding-Related Required Information

Some encoding schemes require specifications beyond the CS and CP elements to complete their definitions. Such specifications are called Additional Coding-related Required Information (ACRI) elements.

Three types of ACRI are defined below.

ACRI PC Mixed Byte (ACRI-PCMB)

This type of ACRI applies to ES values X'2300', X'2305' or X'3300' (see semantics of these ES values in Figure 9). It cannot be specified with any other ES values. It consists of the specification of ranges of valid first bytes of double-bytes associated with particular CS, CP pairs that are used with this encoding scheme. An ACRI-PCMB has the following format:

where N is the number of ranges of valid first bytes, Sk is the starting byte and Ek is the ending byte in the kth range, for all values of k from 1 to N. Sk and Ek are each in the range 128 to 255.

For example, ACRI-PCMB associated with CCSID 00942 in the CCSID Registry (see Appendix C: CCSID Repository ) will be represented as:

2 129 159 224 252

In this example, there are two sets of valid first bytes (shown as their decimal values). The first set of 31 values is in the range 129 to 159 (X'81' to X'9F'), and the second set of 29 values in the range 224 to 252 (X'E0' to X'FC'). Thus a total of 60 double-byte wards (14) can be defined using this ACRI-PCMB.

Other formats, such as a bit-pattern representation, are also possible.

ACRI Type EUC (ACRI-EUC)

This type of ACRI applies to ES value X'4403' only. It specifies the number of coded character sets and the width of each. It has the following format:

where N is the number of coded graphic character sets, and Wn is the width of the nth set. If a G set is not used then the value of W is 0 and the corresponding CS/CP entries will be X'FDFF'.

ACRI Type TCP (ACRI-TCP)

This type of ACRI applies to ES value X'5404' only. It specifies the number of coded character sets followed by a triplet for each consisting of the width of the code points for the set, the length of the escape sequence used to designate the set into G0, and the actual escape sequence. The format is as follows:

n W1 LD1 D1 W2 LD2 D2 ... Wn LDn Dn

where

n =number of CGCSGIDs associated with the CCSID

W =width of code points in the code page

LD =length of the designation escape sequence

D =actual designation sequence

For example, the ACRI-TCP for CCSID 00965 (TCP for Traditional Chinese) is:

03 01 03 ESC 28 42 02 04 ESC 24 29 30 02 04 ESC 24 29 31

In this example the ESC mnemonic is shown, rather than the hex value 1B, to allow for ease of readability.

The format of the Escape Sequences is defined in ISO 2022. The "final byte," which defines the actual coded character set to be used, is defined in the ISO document International Register of Coded Character Sets to be Used With Escape Sequences.

A CCSID is a 16-bit identifier defined by CDRA. A CCSID by definition uniquely defines a data encoding. Given a CCSID tag and a valid code point, the character associated with that code point can be precisely identified. This is because the definition of the CCSID is linked to the definition of the code page in the IBM corporate registry. The definition of the control characters associated with a CCSID are inherited from the defintion of controls defined for the related encoding scheme.

CCSIDs can be defined as growing. A growing CCSID is defined when the related code page is expected to be expanded. When the CCSID grows (ie, more characters are added to the related code page and character set), a non-growing, fixed, CCSID is defined for the existing resources and the growing CCSID takes on the characteristics of the expanded resources. The range of CCSID values is 00000 (X'0000') to 65535 (X'FFFF'). The bit allocations in a CCSID are shown in Figure 10.

Figure 10. Bit Allocations in the Coded Character Set Identifier (CCSID)

Each CCSID has a corresponding long-form identifier or has a predefined special meaning. Figure 11 shows the allocation of CCSID values.

Figure 11. Allocation of CCSID values

1. Interoperable CCSIDs:
   
   Interoperable CCSIDs have the following characteristics:
   • The character set is an interoperable set
   
   Supporting interoperable CCSIDs allows for:
   • Data interchange across various environments within a country/language (16) without data loss
   • Data interchange across various environments and countries within Group 1 without data loss.
   • See Appendix C: CCSID Repository for access to a complete list of CCSIDs.
     


2. Global Use CCSIDs
   
   Some CCSIDs are defined with character sets that are globally applicable. These typically use the Syntactic Character Set (CS 640).
   See Appendix C: CCSID Repository for access to a complete list of CCSIDs.
   


3. Universal
   
   This category encompasses all the encoding forms of UCS, it is a Large Multi-Script Character set covering all the living languages of today, is the character set of the world-wide web and is expected to supported in all computing environments. Its character set is a super set of the character sets of the non-UCS CCSIDs.
   


4. Coexistence and Migration CCSIDs
   
   All other CCSIDs are classified as Coexistence and Migration CCSIDs. They may be widely used within a country or environment but not have the properties of an interoperable CCSID, or they may have a very specific, limited use such as a 7-bit symbols set.
   See Appendix C: CCSID Repository for access to a complete list of CCSIDs.

Tagging in CDRA

When data is tagged with a CCSID, the GCGIDs assigned to the graphic character code points must be those defined by the CCSID.

When a graphic character is represented in data using a CCSID tag:

• It is in one of the CS elements found in one of the CS,CP pairs identified by the CCSID.
• The number of bytes in the code point is defined in the ESID element associated with the CCSID. In the case of mixed encodings, the number of bytes in the code point is defined by the respective CP element.
• The encoding scheme indicates if code extension controls (such as SO and SI) are required.

When data is to be interpreted according to a CCSID value:

• Parsing logic that respects the ESID element of the CCSID tag is needed to correctly process the data based on the number of bytes in each code point.
• The code point should be verified to be located in the graphic character encoding space with an assigned GCGID for the appropriate CP element of the CCSID.
• The GCGID must exist in the character set identified by the appropriate CS element of the CCSID.

When data with no assigned graphic character meaning is found, it should be treated as bytes.

These concepts are explained using two examples.

Example 1: Pure Single-Byte Case

In this example, let ESa, CSa, and CPa (in a single-byte encoding scheme) be the Encoding Scheme, Character Set, and Code Page elements of CCSIDa. See Figure 12.

Figure 12. Meaning of Tagging: A Single Byte Example

The encoding space defined by ESa is composed of Cand G1, where C is the control area and G1 is the graphic area. G1arepresents all code points that have been assigned to the GCGIDs found in character set CSa. Only code points found within G1a can have graphic character meaning according to the definition of CCSIDa. G1mrepresents all code points within CPa that have assigned GCGID values.

Example 2: Case of Mixed Single-Byte Double-Byte in PC

The example shown in Figure 13 uses a PC mixed single-byte and double-byte encoding. Let the elements of CCSIDa be ESa, CSa1, CPa1, CSa2, CPa2, and Fa (=ACRI-PCMB, ranges of valid first bytes).

Figure 13. Meaning of Tagging: A PC Mixed SB/DB Example

The encoding space defined by ESa is composed of C, G1, and G2, where C is the control area, G1 is the single-byte graphic area, and G2 is the double-byte graphic area. G1arepresents all of the single-byte code points that have been assigned to the GCGIDs found in character set CSa1. G2a represents all of the double-byte code points that have been assigned to the GCGIDs found in character set CSa2. Only code points found within G1a or G2a can have graphic character meaning according to the definition of CCSIDa. G1m and G2mrepresent all code points within CPa1 and CPa2, respectively, that have assigned GCGID values. Farepresents the set of valid first bytes for double-byte code points found in G2a.

Meaning of Tagging in CDRA

CDRA has a dependency on other architectures, processes, or functions to provide proper graphic character data processing. The tags can be used to set the meaning or derive the meaning of code points in data to the extent defined above, when:

• Proper validation or filtering mechanisms to separate graphic character data from others are in place
• The CDRA tags are not erroneously applied to interpret the meaning of the bytes that do not have any graphic character meaning.

Relationship of Tags to Data Path

The data along with its tag may traverse many different systems through networks. In the process the tag value may get changed to reflect any conversion of the data. The tag values do not have any relationship to the data path.