Documentation
¶
Overview ¶
Package transcode converts data between character sets, including the single-byte EBCDIC code pages used on mainframes.
Translation tables ¶
A Table translates data byte by byte, without going through Unicode, so it reproduces exactly an existing single-byte translation. It is read from and written to its binary form (256 bytes) or its text form (512 hexadecimal digits), can be inverted when it is bijective, and composed with another table.
Code pages ¶
A CodePage is a single-byte character set: each byte stands for at most one Unicode character. It implements golang.org/x/text/encoding.Encoding, so it converts to and from UTF-8, and through UTF-8 to any other encoding. A page is built from the character of each byte (NewCodePage), read from a mapping file in the format of the Unicode consortium (ParseCodePage), or taken from golang.org/x/text/encoding/charmap (FromCharmap).
TableBetween computes once the Table that converts data between two code pages. A byte without equivalent in the target page makes it fail, the error naming every such byte, unless a substitution byte is given.
Registry ¶
A Registry holds the character sets and translation tables available by name. It comes with 121 built-in character sets: the Unicode encodings, the single-byte code pages and the multi-byte encodings of golang.org/x/text/encoding, and 60 single-byte EBCDIC code pages, 57 of which are generated from the tables that IBM publishes through the ICU project. Code pages and tables can be added at run time; the registry lists them all, for instance for an administration screen.
Records ¶
Converting a whole file also converts its record separators: the line feed of ASCII becomes 0x25 in EBCDIC, so that a program which cuts the file into lines at 0x0A no longer finds them. A RecordTransformer applies a conversion to the content of each record instead, and lays the records out as asked. Each side has its RecordFormat:
- Stream: the data is a single record;
- Separated: records ended by a separator, such as LF, CRLF or the EBCDIC new line NL;
- SameSeparators, on output: the separators read on input, unchanged;
- Fixed: records of a fixed length, padded on output and optionally trimmed on input;
- RDW: variable-length records preceded by a record descriptor word, as z/OS writes them.
Combining them adds, removes or changes separators: Separated input and Fixed output turn lines into fixed-length records, Fixed input and Separated output the reverse.
Conversions ¶
Registry.NewConverter builds the conversion that a Conversion describes by names: from a source to a target character set, UTF-8 to EBCDIC for instance, or with a translation table, as a whole or record by record. Separators and padding are then given as characters, which the converter writes in each character set: LF becomes 0x25 in EBCDIC, NEL the EBCDIC new line 0x15, and padding defaults to the space of the target. Records too long are cut between two characters, and byte order marks are read and written once. Without records, a conversion behaves exactly as the decoder and encoder of golang.org/x/text.
With the same character set as source and target, a conversion only lays the records out again: their content is copied byte for byte, never decoded, so that a byte which stands for no character goes through unchanged. Separators and padding are still written in that character set, and a record too long is still cut between two characters.
Streams ¶
Every conversion implements golang.org/x/text/transform.Transformer, so it applies to a stream of any size through golang.org/x/text/transform.NewReader or golang.org/x/text/transform.NewWriter.
Index ¶
- Constants
- Variables
- type CodePage
- func (p *CodePage) Byte(char rune) (byte, bool)
- func (p *CodePage) Char(b byte) (rune, bool)
- func (p *CodePage) Chars() [TableSize]rune
- func (p *CodePage) IsDecodeOnly(b byte) bool
- func (p *CodePage) MarshalText() ([]byte, error)
- func (p *CodePage) Name() string
- func (p *CodePage) NewDecoder() *encoding.Decoder
- func (p *CodePage) NewEncoder() *encoding.Encoder
- func (p *CodePage) Replacement() byte
- func (p *CodePage) String() string
- type Conversion
- type Converter
- type Entry
- type Kind
- type PageOption
- type RecordFormat
- type RecordTransformer
- type Registry
- func (r *Registry) CodePage(name string) (*CodePage, error)
- func (r *Registry) Encoding(name string) (encoding.Encoding, error)
- func (r *Registry) List() []Entry
- func (r *Registry) Lookup(name string) (Entry, bool)
- func (r *Registry) NewConverter(conversion *Conversion) (*Converter, error)
- func (r *Registry) Remove(name string) error
- func (r *Registry) SetCodePage(page *CodePage) error
- func (r *Registry) SetTable(name string, table *Table) error
- func (r *Registry) Table(name string) (Table, error)
- type Table
- func (t *Table) Invert() (Table, error)
- func (t *Table) MarshalBinary() ([]byte, error)
- func (t *Table) MarshalText() ([]byte, error)
- func (*Table) Reset()
- func (t *Table) Then(next *Table) Table
- func (t *Table) Transform(dst, src []byte, _ bool) (nDst, nSrc int, err error)
- func (t *Table) Translate(p []byte)
- func (t *Table) UnmarshalBinary(data []byte) error
- func (t *Table) UnmarshalText(text []byte) error
- type TableOption
- type UnsupportedCharError
Examples ¶
Constants ¶
const ( // LF is the line feed, which ends lines on Unix and Linux systems. LF = "\n" // CRLF is the carriage return and line feed, which end lines on Windows. CRLF = "\r\n" // NL is the EBCDIC new line, which ends lines in the z/OS UNIX file system. NL = "\x15" // EBCDICLF is the EBCDIC line feed, into which EBCDIC code pages turn LF. EBCDICLF = "\x25" )
Record separators commonly found in data.
const NEL = "\u0085"
NEL is the Unicode next line character. EBCDIC code pages turn it into their new line, 0x15, and LF into their line feed, 0x25: in the records of a Conversion, NEL stands for the EBCDIC new line.
const NoChar rune = -1
NoChar marks, among the characters given to NewCodePage, a byte that stands for no character.
const TableSize = 256
TableSize is the number of entries of a Table, one per byte value.
Variables ¶
var ( // ErrInvalidCodePage is returned when a code page definition is invalid. ErrInvalidCodePage = errors.New("invalid code page") // ErrUndefinedByte reports a byte that stands for no character in its // code page. ErrUndefinedByte = errors.New("byte stands for no character") // ErrUnsupportedChar reports a character that a code page cannot // represent. ErrUnsupportedChar = errors.New("character not supported by the code page") )
var ( // ErrRecordFormat is returned for an invalid record format, or one that // does not apply to the direction it is used in. ErrRecordFormat = errors.New("invalid record format") // ErrInvalidRecord is returned for a record that cannot be read or // written as its format requires. ErrInvalidRecord = errors.New("invalid record") )
var ( // ErrUnknownName is returned for a name that the registry does not hold. ErrUnknownName = errors.New("unknown character set or table") // ErrWrongKind is returned when an entry is not of the kind asked for. ErrWrongKind = errors.New("entry of another kind") // ErrInvalidName is returned for an empty name, or a name that starts or // ends with white space. ErrInvalidName = errors.New("invalid name") // ErrNameTaken is returned when a name, or a name that the registry // takes for the same, is used by another entry. ErrNameTaken = errors.New("name already used by another entry") // ErrBuiltinEntry is returned on an attempt to change or remove a // built-in entry. ErrBuiltinEntry = errors.New("built-in entry cannot be changed") )
var ( // ErrInvalidTable is returned when data cannot be read as a [Table]. ErrInvalidTable = errors.New("invalid translation table") // ErrNotInvertible is returned by [Table.Invert] when the table // translates two bytes into the same value. ErrNotInvertible = errors.New("translation table is not invertible") )
ErrConversion is returned for a conversion that cannot be made as described.
Functions ¶
This section is empty.
Types ¶
type CodePage ¶
type CodePage struct {
// contains filtered or unexported fields
}
CodePage is a single-byte character set: each byte stands for at most one Unicode character.
*CodePage implements encoding.Encoding. Its decoder, which converts the page to UTF-8, turns a byte that stands for no character into U+FFFD. Its encoder, which converts UTF-8 to the page, stops with an *UnsupportedCharError on a character that the page cannot represent; encoding.ReplaceUnsupported turns that error into the page's substitution byte. Both follow the conventions of golang.org/x/text/encoding/charmap.
A CodePage is immutable and safe for concurrent use.
func FromCharmap ¶
FromCharmap returns the code page that stands for the same characters as a code page of golang.org/x/text/encoding/charmap, under the same name. Bytes that the charmap decodes into U+FFFD stand for no character.
func NewCodePage ¶
NewCodePage returns the code page called name in which byte b stands for chars[b], or for no character when chars[b] is NoChar.
When several bytes stand for the same character, the encoder uses the lowest one that is not marked DecodeOnly. U+FFFD cannot be assigned to a byte: the decoder reserves it for bytes that stand for no character.
func ParseCodePage ¶
ParseCodePage reads the code page called name from a mapping file, the format of the Unicode consortium mapping tables. Each line maps a byte to a character, both in hexadecimal with a 0x prefix, separated by spaces or tabs:
0xC1 0x0041 # LATIN CAPITAL LETTER A
A line with a byte and no character, and a byte without a line, stand for no character. Text after # is a comment; blank lines are ignored; lines may end with LF or CRLF. A byte may appear only once, and a byte cannot stand for a sequence of characters.
The comment "decode only" on a mapping line marks its byte DecodeOnly:
0x51 0x0E48 # decode only
func (*CodePage) Byte ¶
Byte returns the byte that stands for char, and false when the page cannot represent char.
func (*CodePage) Char ¶
Char returns the character that b stands for, and false when b stands for no character.
func (*CodePage) Chars ¶
Chars returns the character of every byte, NoChar marking the bytes that stand for no character.
func (*CodePage) IsDecodeOnly ¶
IsDecodeOnly reports whether b is marked DecodeOnly: the decoder reads it, the encoder never writes it.
func (*CodePage) MarshalText ¶
MarshalText returns the mapping file of the page, in the format read by ParseCodePage: one line per byte that stands for a character.
func (*CodePage) NewDecoder ¶
NewDecoder returns a decoder that converts the page to UTF-8.
func (*CodePage) NewEncoder ¶
NewEncoder returns an encoder that converts UTF-8 to the page.
func (*CodePage) Replacement ¶
Replacement returns the byte that encoding.ReplaceUnsupported writes in place of a character that the page cannot represent: the page's SUB control character if it has one, its question mark otherwise.
type Conversion ¶
type Conversion struct {
// Source and Target are the names of the character sets of the data and
// of the result. When they name the same character set, the content is
// copied as it is, byte for byte, without being decoded: only the records
// are laid out again, their separators and padding written in that
// character set.
Source, Target string
// Table is the name of a translation table that converts the data byte by
// byte, in place of Source and Target. These are then single-byte code
// pages that only serve to write separators and padding, and may be left
// empty: separators and padding are then taken as bytes, and padding has
// no default.
Table string
// Input and Output are the record formats of the data and of the result;
// their zero value converts the data as a whole. Their separators and
// padding are characters, which the converter writes in Source and
// Target, with code units of the right size; padding defaults to a space.
// On output, a record too long is cut between two characters.
Input, Output RecordFormat
// Substitute writes the substitution character of Target in place of a
// character that Target cannot represent, instead of failing.
Substitute bool
}
Conversion describes a conversion made by Registry.NewConverter.
type Converter ¶
type Converter struct {
transform.Transformer
}
Converter converts data as a Conversion describes it. It implements transform.Transformer.
type Entry ¶
type Entry struct {
// Name is the name of the entry.
Name string
// Kind tells what the entry is.
Kind Kind
// EBCDIC reports a single-byte code page of the EBCDIC family.
EBCDIC bool
// Builtin reports an entry of the library, which cannot be changed or
// removed.
Builtin bool
// Deprecated reports a built-in entry kept so that existing settings
// still work, which a list offered for new settings should leave out.
Deprecated bool
// Source tells where the definition of a built-in entry comes from.
Source string
}
Entry describes a character set or a translation table of a Registry.
type Kind ¶
type Kind int
Kind tells what a registry entry is.
const ( // KindUnicode is a Unicode encoding: UTF-8, UTF-16 or UTF-32. KindUnicode Kind = iota + 1 // KindSingleByte is a single-byte code page, also available as a // [CodePage]. KindSingleByte // KindMultiByte is a multi-byte encoding, such as Shift JIS or GB18030. KindMultiByte // KindTable is a byte-to-byte translation [Table]. KindTable )
Kinds of registry entries.
type PageOption ¶
type PageOption func(*CodePage)
PageOption adjusts a code page built by NewCodePage.
func DecodeOnly ¶
func DecodeOnly(octets ...byte) PageOption
DecodeOnly marks bytes that the decoder reads but the encoder never writes. Some pages place a character at two positions: both decode into the character, which encodes into the other position only.
type RecordFormat ¶
type RecordFormat struct {
// contains filtered or unexported fields
}
RecordFormat describes how data is cut into records, on input, and laid out, on output. The zero value is Stream: the data is a single record.
A RecordFormat is built by Stream, Separated, SameSeparators, Fixed or RDW, then adjusted with its methods. Separators and padding are bytes of the data, in its encoding: the EBCDIC space is "\x40", the UTF-16LE line feed "\n\x00".
func Fixed ¶
func Fixed(length int) RecordFormat
Fixed is the format of records of length bytes, with nothing between them. On input, the data must end with a whole record. On output, a shorter record is completed with the padding given by RecordFormat.WithPad, and a longer one makes the conversion fail, unless RecordFormat.Truncating.
func RDW ¶
func RDW() RecordFormat
RDW is the format of variable-length records preceded by a record descriptor word, as z/OS writes them: 4 bytes, the length of the record header included as a 2-byte big-endian number, then 2 zero bytes. Spanned records and block descriptor words are not supported. On output, a record holds at most 32,756 bytes of content.
func SameSeparators ¶
func SameSeparators() RecordFormat
SameSeparators is an output format only, for Separated input: each record is followed by the separator that ended it on input, unchanged. The separators thus stay out of the conversion, which applies to the content of the records only.
func Separated ¶
func Separated(separators ...string) RecordFormat
Separated is the format of records ended by a separator. On input, a record ends with any of the separators, the first that occurs, and the longest at the same place, so that Separated(CRLF, LF) reads both kinds of lines; the last record may lack its separator. On output, each record is followed by the first separator, except a last record that had none on input.
func Stream ¶
func Stream() RecordFormat
Stream is the format of data that is not cut into records: on input, the data is a single record; on output, records are written one after the other, with nothing between them.
func (RecordFormat) TrimmingPad ¶
func (f RecordFormat) TrimmingPad() RecordFormat
TrimmingPad makes the input of fixed-length or RDW records remove the padding repeated at the end of each record.
func (RecordFormat) Truncating ¶
func (f RecordFormat) Truncating() RecordFormat
Truncating makes the output of fixed-length or RDW records cut a record that is too long, instead of failing. The cut falls at a multiple of the code unit; a Converter makes it fall between two characters.
func (RecordFormat) WithPad ¶
func (f RecordFormat) WithPad(pad string) RecordFormat
WithPad sets the padding of fixed-length records, one code unit long: on output, it completes short records; on input, RecordFormat.TrimmingPad removes it from the end of records.
func (RecordFormat) WithUnit ¶
func (f RecordFormat) WithUnit(size int) RecordFormat
WithUnit sets the size of the code units of the data: 1 (the default), 2 for UTF-16 or 4 for UTF-32. Separators are only recognized at the start of a code unit, so that a byte sequence that straddles two characters is never taken for one; lengths are multiples of the unit.
type RecordTransformer ¶
type RecordTransformer struct {
// contains filtered or unexported fields
}
RecordTransformer converts data record by record; it implements transform.Transformer. It streams the content of separated and stream records, and holds a whole record only when its format bounds its length: fixed-length and RDW input, fixed-length and RDW output.
func NewRecordTransformer ¶
func NewRecordTransformer( input RecordFormat, convert transform.Transformer, output RecordFormat, ) (*RecordTransformer, error)
NewRecordTransformer returns a transformer that reads records laid out as input, converts the content of each record with convert, and writes the records laid out as output. A nil convert leaves the content unchanged.
The content of each record is converted on its own: convert is reset at the start of each record and receives its end as the end of the data. With an encoding that writes a byte order mark, the mark is thus written once per record; a Converter writes it once per file instead.
Errors on a record name its number, counted from 1.
func (*RecordTransformer) Reset ¶
func (t *RecordTransformer) Reset()
Reset implements transform.Transformer.
func (*RecordTransformer) Transform ¶
func (t *RecordTransformer) Transform(dst, src []byte, atEOF bool) (nDst, nSrc int, err error)
Transform implements transform.Transformer.
type Registry ¶
type Registry struct {
// contains filtered or unexported fields
}
Registry holds the character sets and translation tables available by name: the built-in ones, and those registered at run time with Registry.SetCodePage and Registry.SetTable. It keeps everything in memory and stores nothing; an application that keeps its own pages and tables registers them when it starts.
A name is looked up as written, then regardless of case, spaces, hyphens and underscores ("utf8" finds "UTF-8", "iso-8859-15" finds "ISO 8859-15"), and IBM code pages also by number ("IBM1147", "CP037", "IBM-01147"). A registered name may not be taken for the name of another entry in these ways.
A Registry is safe for concurrent use.
Example ¶
package main
import (
"fmt"
"code.waarp.fr/lib/transcode"
)
func main() {
registry := transcode.NewRegistry()
// Names are found regardless of case and punctuation, and IBM code
// pages by number.
entry, _ := registry.Lookup("ibm-1147")
fmt.Println(entry.Name, "-", entry.Kind)
page, err := registry.CodePage(entry.Name)
if err != nil {
fmt.Println(err)
return
}
encoded, _ := page.NewEncoder().String("été")
fmt.Printf("% X\n", encoded)
}
Output: IBM Code Page 1147 - single-byte code page C0 A3 C0
func NewRegistry ¶
func NewRegistry() *Registry
NewRegistry returns a registry that holds the built-in character sets.
func (*Registry) CodePage ¶
CodePage returns the single-byte code page called name.
func (*Registry) Encoding ¶
Encoding returns the character set called name. For the single-byte code pages and the encodings that come from golang.org/x/text, it returns the object of golang.org/x/text itself, so that they behave exactly as with that package.
func (*Registry) List ¶
List returns every entry: the built-in ones, Unicode encodings first, then single-byte code pages, EBCDIC pages by number, and multi-byte encodings; then the registered ones, by name.
func (*Registry) Lookup ¶
Lookup returns the entry called name.
func (*Registry) NewConverter ¶
func (r *Registry) NewConverter(conversion *Conversion) (*Converter, error)
NewConverter returns the converter that makes conversion.
Without records, a conversion between two character sets is the chain of the decoder of Source and the encoder of Target, as with golang.org/x/text. With records, the content of each record is converted on its own, and a byte order mark is read or written once, at the start of the data: on input, a mark of the other byte order fails.
Example ¶
package main
import (
"fmt"
"code.waarp.fr/lib/transcode"
"golang.org/x/text/transform"
)
func main() {
registry := transcode.NewRegistry()
// UTF-8 lines to the French EBCDIC page 1147, keeping the line feeds for
// a reader that cuts the result into lines at 0x0A.
converter, err := registry.NewConverter(&transcode.Conversion{
Source: "UTF-8",
Target: "IBM1147",
Input: transcode.Separated(transcode.LF),
Output: transcode.SameSeparators(),
})
if err != nil {
fmt.Println(err)
return
}
converted, _, _ := transform.String(converter, "été\nhiver\n")
fmt.Printf("% X\n", converted)
}
Output: C0 A3 C0 0A 88 89 A5 85 99 0A
func (*Registry) Remove ¶
Remove removes the registered entry called name.
func (*Registry) SetCodePage ¶
SetCodePage registers page under its name, or replaces the page registered under that name.
func (*Registry) SetTable ¶
SetTable registers a copy of table under name, or replaces the table registered under that name.
type Table ¶
Table is a single-byte translation table: byte b is translated into t[b].
A table converts data byte by byte, without going through Unicode. It can therefore reproduce exactly any existing single-byte translation, including the handling of control characters and of bytes that a character set leaves undefined.
A table has two external forms, both read by ParseTable: the binary form, 256 bytes where the byte at offset b is the translation of b, and the text form, 512 hexadecimal digits, two per byte in the same order.
The zero value translates every byte into 0x00; IdentityTable returns a table that leaves data unchanged. *Table implements transform.Transformer; a table must not be modified while a transformation uses it.
func IdentityTable ¶
func IdentityTable() Table
IdentityTable returns the table that translates every byte into itself.
func ParseTable ¶
ParseTable reads a table from its binary form, which is exactly 256 bytes long, or from its text form, described in Table.UnmarshalText.
Example ¶
package main
import (
"fmt"
"code.waarp.fr/lib/transcode"
"golang.org/x/text/transform"
)
func main() {
// A table in text form that turns lower-case ASCII letters into
// upper-case ones and leaves every other byte unchanged.
upper := transcode.IdentityTable()
for b := byte('a'); b <= 'z'; b++ {
upper[b] = b - 'a' + 'A'
}
text, _ := upper.MarshalText()
table, err := transcode.ParseTable(text)
if err != nil {
fmt.Println(err)
return
}
converted, _, _ := transform.String(&table, "Hello, world")
fmt.Println(converted)
}
Output: HELLO, WORLD
func TableBetween ¶
func TableBetween(source, target *CodePage, options ...TableOption) (Table, error)
TableBetween returns the table that converts data from the source code page to the target code page: each byte is translated into the byte that stands for the same character in the target page. The table is computed once, then converts data byte by byte.
A byte has no equivalent when it stands for no character in the source page (ErrUndefinedByte), or when the target page cannot represent its character (ErrUnsupportedChar). TableBetween then fails, and the error names every such byte, unless WithSubstitute is given.
func (*Table) Invert ¶
Invert returns the table that undoes t: applying t, then the result, leaves data unchanged. It fails with ErrNotInvertible when t translates two bytes into the same value, since the reverse translation would then be ambiguous; the error names the conflicting bytes.
func (*Table) MarshalBinary ¶
MarshalBinary returns the binary form of t: 256 bytes, the byte at offset b being the translation of b.
func (*Table) MarshalText ¶
MarshalText returns the text form of t: 512 upper-case hexadecimal digits, two per byte in byte order, on a single line.
func (*Table) Reset ¶
func (*Table) Reset()
Reset implements transform.Transformer. A table keeps no state.
func (*Table) Then ¶
Then returns the table that applies t, then next: byte b is translated into next[t[b]].
func (*Table) Transform ¶
Transform implements transform.Transformer. It translates as many bytes of src as dst can hold, and returns transform.ErrShortDst when dst is shorter than src.
func (*Table) UnmarshalBinary ¶
UnmarshalBinary reads the binary form of a table, which must be exactly 256 bytes long. On error, t is left unchanged.
func (*Table) UnmarshalText ¶
UnmarshalText reads the text form of a table: 512 hexadecimal digits, two per byte in byte order. Digits may be upper or lower case. Spaces, tabs and line breaks between digits are ignored, so the digits may be split over several lines. A hyphen that ends a line, which configuration exports use to continue a long value on the next line, is ignored too:
00010203372D2E2F1605250B0C0D0E0F - 101112133C3D322618193F271C1D1E1F -
On error, t is left unchanged.
type TableOption ¶
type TableOption func(*tableSettings)
TableOption adjusts how TableBetween builds a table.
func WithSubstitute ¶
func WithSubstitute(substitute byte) TableOption
WithSubstitute makes TableBetween translate into substitute every byte that has no equivalent in the target page, instead of failing.
type UnsupportedCharError ¶
type UnsupportedCharError struct {
// Page is the name of the code page.
Page string
// Char is the character that the page cannot represent.
Char rune
// contains filtered or unexported fields
}
UnsupportedCharError reports a character that a code page cannot represent. Invalid UTF-8 is reported as U+FFFD. The error matches ErrUnsupportedChar with errors.Is, and encoding.ReplaceUnsupported replaces the character with the page's substitution byte.
func (*UnsupportedCharError) Error ¶
func (e *UnsupportedCharError) Error() string
func (*UnsupportedCharError) Replacement ¶
func (e *UnsupportedCharError) Replacement() byte
Replacement returns the page's substitution byte, which encoding.ReplaceUnsupported writes in place of the character.
func (*UnsupportedCharError) Unwrap ¶
func (*UnsupportedCharError) Unwrap() error
Unwrap returns ErrUnsupportedChar.
Source Files
¶
- between.go
- builtin.go
- codepage.go
- converter.go
- doc.go
- ebcdic_tables.go
- mapfile.go
- records.go
- registry.go
- table.go
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
gen
command
Command gen generates ebcdic_tables.go, the EBCDIC code pages built into the transcode package, from the mapping files of the ICU project.
|
Command gen generates ebcdic_tables.go, the EBCDIC code pages built into the transcode package, from the mapping files of the ICU project. |