Encode to EBCDIC doesn't take into account conversion table irregularities

[VladimirFilippov]

BPO	30586
Nosy	@vstinner, @ezio-melotti, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2017-06-07.11:41:40.052>
labels = ['type-bug', 'invalid', 'expert-unicode']
title = "Encode to EBCDIC doesn't take into account conversion table irregularities"
updated_at = <Date 2017-06-07.15:45:17.575>
user = 'https://bugs.python.org/VladimirFilippov'

bugs.python.org fields:

activity = <Date 2017-06-07.15:45:17.575>
actor = 'Vladimir Filippov'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Unicode']
creation = <Date 2017-06-07.11:41:40.052>
creator = 'Vladimir Filippov'
dependencies = []
files = []
hgrepos = []
issue_num = 30586
keywords = []
message_count = 3.0
messages = ['295329', '295336', '295352']
nosy_count = 4.0
nosy_names = ['vstinner', 'ezio.melotti', 'serhiy.storchaka', 'Vladimir Filippov']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue30586'
versions = ['Python 3.6']

[VladimirFilippov]

These 4 symbols were encoded incorrectly to EBCDIC (codec cp500): "![]|". Correct table of conversation for these symbols described in https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.3.0/com.ibm.swg.im.iis.ds.parjob.adref.doc/topics/r_deeadvrf_Conversion_Table_Irregularities.html

This code:
--------------------
ascii = '![]|';
print("ASCII: " + bytes(ascii, 'ascii').hex())
res = ascii.encode('cp500')
print ("EBCDIC: " +res.hex())
--------------------
on Python 3.6.1 produce this output:
--------------------
ASCII: 215b5d7c
EBCDIC: 4f4a5abb
--------------------

Expected encoding (from IBM's table):
! - 5A
[ - AD
] - BD
| - 4F

Workaround: use this translation after encoding
bytes.maketrans(b'\x4F\x4A\x5A\xBB', b'\x5A\xAD\xBD\x4F')

[serhiy-storchaka]

The cp500 codec in Python is generated from the table ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP500.TXT .

There are several EBCDIC code pages. EBCDIC-compatible encodings supported in Python are: cp037, cp273, cp424, cp500, cp875, cp1026 and cp1140. Three of them, cp037, cp424 and cp1140, encode '!' to b'\x5A' and '|' to b'\x4F'.

[VladimirFilippov]

According to ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP037.TXT symbols [ and ] have other codes (instead of 0xAD and 0xBD):
0xBA 0x005B #LEFT SQUARE BRACKET
0xBB 0x005D #RIGHT SQUARE BRACKET

Looks like ftp://ftp.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/EBCDIC/CP500.TXT was created based on https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.3.0/com.ibm.swg.im.iis.ds.parjob.adref.doc/topics/r_deeadvrf_ASCII_to_EBCDIC.html
But this information "This translation is not bidirectional. Some EBCDIC characters cannot be translated to ASCII and some conversion irregularities exist in the table. For more information, see Conversion table irregularities." was ignored. Additional, this line from CP500.TXT:
0xBB 0x007C #VERTICAL LINE
haven't any source in IBM's table.

Example from z/OS mainframe:
-------------------
bash-4.3$ iconv -f 819 -t 1047 -T ascii.txt > ebcdic.txt
bash-4.3$ ls -T *.txt
t ISO8859-1 T=on ascii.txt
t IBM-1047 T=on ebcdic.txt
bash-4.3$ cat ascii.txt
![]|bash-4.3$ od -h ascii.txt
0000000000 21 5B 5D 7C
0000000004
bash-4.3$ cat ebcdic.txt
![]|bash-4.3$ od -h ebcdic.txt
0000000000 5A AD BD 4F
0000000004
-------------------

[serhiy-storchaka]

The iconv-based codec engine added in #152997 (PR #153001) offers a way around this: where IBM-exact EBCDIC conversion matters, iconv:CP1047 (a.k.a. IBM1047) uses the platform's own table instead of the built-in cp1047.

The built-in codec itself is unchanged.