• 1.  Codepage conversion issue

    Posted 21 days ago

    So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

    Here are the changes I made:

    • In project micro:
      • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
      • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
    • In project https://github.com/zyedidia/terminal:
      • Added zos build tags to ioctl_posix.go and vt_posix.go.
    • In https://github.com/zyedidia/tcell:
      • Created tscreen_zos.go from tscreen_solaris.go.
    • In project https://github.com/creack/pty: I
      • Added the !zos build tag to pty_unsupported.go
      • Added new file pty_zos.go, which is as follows:
    //go:build zos
    // +build zos
    package pty
    import (
    func open() (pty, tty *os.File, err error) {
            ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)
            if err != nil {
                    return nil, nil, err
            p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")
            if p == nil {
                    return nil, nil, err
            // In case of error after this point, make sure we close the ptmx fd.
            defer func() {
                    if err != nil {
                            _ = p.Close() // Best effort.
            sname, err := unix.Ptsname(ptmxfd)
            if err != nil {
                    return nil, nil, err
            _, err = unix.Grantpt(ptmxfd)
            if err != nil {
                    return nil, nil, err
            if _, err = unix.Unlockpt(ptmxfd); err != nil {
                    return nil, nil, err
            ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)
            if err != nil {
                    return nil, nil, err
            t := os.NewFile(uintptr(ptsfd), sname)
            if err != nil {
                    return nil, nil, err
            return p, t, nil

    The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

    Frank Swarbrick

  • 2.  RE: Codepage conversion issue

    Posted 20 days ago
    Edited by Hyder Naqvi 20 days ago

    Hello Frank that's an interesting problem.
    One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
    Here's a codepage conversion example.

    Please let me know your thoughts on using go-recordio as a solution.

    Hyder Naqvi

  • 3.  RE: Codepage conversion issue

    Posted 20 days ago

    Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

    func Getsize(t *os.File) (rows, cols int, err error)
    func InheritSize(pty, tty *os.File) error
    func Open() (pty, tty *os.File, err error)
    func Setsize(t *os.File, ws *Winsize) error
    func Start(cmd *exec.Cmd) (*os.File, error)
    func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)
    func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)
    func GetsizeFull(t *os.File) (size *Winsize, err error)

    It's not involved with the reads and writes themselves.

    Frank Swarbrick

  • 4.  RE: Codepage conversion issue

    Posted 16 days ago

    So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

    diff --git a/src/pty.c b/src/pty.c
    index f11a22dcd..b8dea655f 100644
    --- a/src/pty.c
    +++ b/src/pty.c
    @@ -194,6 +194,12 @@ mch_openpty(char **ttyn)
         if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1)
     	return -1;
    +#ifdef __MVS__
    +    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged
    +    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};
    +    fcntl(f, F_CONTROL_CVT, &cvtreq);
         // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and
         // exec()s pt_chmod
         sigcld = mch_signal(SIGCHLD, SIG_DFL);

    I first made some changes to golang.org/x/sys/unix:

    diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.go
    index 312ae6a..84fefae 100644
    --- a/unix/syscall_zos_s390x.go
    +++ b/unix/syscall_zos_s390x.go
    @@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) {
     //sys  Fchown(fd int, uid int, gid int) (err error)
     //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A
     //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL
    +//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL
     //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC
     //sys  fstat(fd int, stat *Stat_LE_t) (err error)
     //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A
    @@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {
    --- a/unix/ztypes_zos_s390x.go
    +++ b/unix/ztypes_zos_s390x.go
    @@ -377,6 +377,12 @@ type Flock_t struct {
            Pid    int32
    +type F_cnvrt struct {
    +       Cvtcmd int32
    +       Pccsid int16
    +       Fccsid int16
     type Termios struct {
            Cflag uint32
            Iflag uint32

    Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

            // Needed for z/OS so that the characters are not garbled if ptyp* is untagged
            cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}
            if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {
                    return nil, nil, err

    There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

    VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

    By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

    Frank Swarbrick

  • 5.  RE: Codepage conversion issue

    Posted 15 days ago

    Frank, you should retrieve the tag on the file descriptor and check its value and text bit and only apply fcntl if it is really needed.

    Cw Cheung

  • 6.  RE: Codepage conversion issue

    Posted 15 days ago

    Yeah, I was wondering about that.  Since vimport didn't have it, I chose not to.  But you're probably right.

    Frank Swarbrick