Go

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

diff --git a/src/pty.c b/src/pty.cindex f11a22dcd..b8dea655f 100644--- a/src/pty.c+++ b/src/pty.c@@ -194,6 +194,12 @@ mch_openpty(char **ttyn)     if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1) 	return -1; +#ifdef __MVS__+    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged+    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};+    fcntl(f, F_CONTROL_CVT, &cvtreq);+#endif+     // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and     // exec()s pt_chmod     sigcld = mch_signal(SIGCHLD, SIG_DFL);

I first made some changes to golang.org/x/sys/unix:

diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.goindex 312ae6a..84fefae 100644--- a/unix/syscall_zos_s390x.go+++ b/unix/syscall_zos_s390x.go@@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) { //sys  Fchown(fd int, uid int, gid int) (err error) //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL+//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC //sys  fstat(fd int, stat *Stat_LE_t) (err error) //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A@@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {        return }--- a/unix/ztypes_zos_s390x.go+++ b/unix/ztypes_zos_s390x.go@@ -377,6 +377,12 @@ type Flock_t struct {        Pid    int32 }+type F_cnvrt struct {+       Cvtcmd int32+       Pccsid int16+       Fccsid int16+}+ type Termios struct {        Cflag uint32        Iflag uint32

Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

        // Needed for z/OS so that the characters are not garbled if ptyp* is untagged        cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}        if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {                return nil, nil, err        }

There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Frank, you should retrieve the tag on the file descriptor and check its value and text bit and only apply fcntl if it is really needed.

So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

diff --git a/src/pty.c b/src/pty.cindex f11a22dcd..b8dea655f 100644--- a/src/pty.c+++ b/src/pty.c@@ -194,6 +194,12 @@ mch_openpty(char **ttyn)     if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1) 	return -1; +#ifdef __MVS__+    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged+    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};+    fcntl(f, F_CONTROL_CVT, &cvtreq);+#endif+     // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and     // exec()s pt_chmod     sigcld = mch_signal(SIGCHLD, SIG_DFL);

I first made some changes to golang.org/x/sys/unix:

diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.goindex 312ae6a..84fefae 100644--- a/unix/syscall_zos_s390x.go+++ b/unix/syscall_zos_s390x.go@@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) { //sys  Fchown(fd int, uid int, gid int) (err error) //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL+//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC //sys  fstat(fd int, stat *Stat_LE_t) (err error) //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A@@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {        return }--- a/unix/ztypes_zos_s390x.go+++ b/unix/ztypes_zos_s390x.go@@ -377,6 +377,12 @@ type Flock_t struct {        Pid    int32 }+type F_cnvrt struct {+       Cvtcmd int32+       Pccsid int16+       Fccsid int16+}+ type Termios struct {        Cflag uint32        Iflag uint32

Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

        // Needed for z/OS so that the characters are not garbled if ptyp* is untagged        cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}        if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {                return nil, nil, err        }

There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Yeah, I was wondering about that.  Since vimport didn't have it, I chose not to.  But you're probably right.

Frank, you should retrieve the tag on the file descriptor and check its value and text bit and only apply fcntl if it is really needed.

So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

diff --git a/src/pty.c b/src/pty.cindex f11a22dcd..b8dea655f 100644--- a/src/pty.c+++ b/src/pty.c@@ -194,6 +194,12 @@ mch_openpty(char **ttyn)     if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1) 	return -1; +#ifdef __MVS__+    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged+    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};+    fcntl(f, F_CONTROL_CVT, &cvtreq);+#endif+     // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and     // exec()s pt_chmod     sigcld = mch_signal(SIGCHLD, SIG_DFL);

I first made some changes to golang.org/x/sys/unix:

diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.goindex 312ae6a..84fefae 100644--- a/unix/syscall_zos_s390x.go+++ b/unix/syscall_zos_s390x.go@@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) { //sys  Fchown(fd int, uid int, gid int) (err error) //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL+//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC //sys  fstat(fd int, stat *Stat_LE_t) (err error) //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A@@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {        return }--- a/unix/ztypes_zos_s390x.go+++ b/unix/ztypes_zos_s390x.go@@ -377,6 +377,12 @@ type Flock_t struct {        Pid    int32 }+type F_cnvrt struct {+       Cvtcmd int32+       Pccsid int16+       Fccsid int16+}+ type Termios struct {        Cflag uint32        Iflag uint32

Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

        // Needed for z/OS so that the characters are not garbled if ptyp* is untagged        cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}        if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {                return nil, nil, err        }

There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Frank, it is ok for you to do a PR against golang.org/x/sys, the usually practice is that you also add a test case to syscall_zos_test.go for the new functions you added. run `go run mksyscall_zos_s390x.go` on a 3.1 system to recreate zsyscall_zos_s390x.go. 

Yeah, I was wondering about that.  Since vimport didn't have it, I chose not to.  But you're probably right.

Frank, you should retrieve the tag on the file descriptor and check its value and text bit and only apply fcntl if it is really needed.

So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

diff --git a/src/pty.c b/src/pty.cindex f11a22dcd..b8dea655f 100644--- a/src/pty.c+++ b/src/pty.c@@ -194,6 +194,12 @@ mch_openpty(char **ttyn)     if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1) 	return -1; +#ifdef __MVS__+    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged+    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};+    fcntl(f, F_CONTROL_CVT, &cvtreq);+#endif+     // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and     // exec()s pt_chmod     sigcld = mch_signal(SIGCHLD, SIG_DFL);

I first made some changes to golang.org/x/sys/unix:

diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.goindex 312ae6a..84fefae 100644--- a/unix/syscall_zos_s390x.go+++ b/unix/syscall_zos_s390x.go@@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) { //sys  Fchown(fd int, uid int, gid int) (err error) //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL+//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC //sys  fstat(fd int, stat *Stat_LE_t) (err error) //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A@@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {        return }--- a/unix/ztypes_zos_s390x.go+++ b/unix/ztypes_zos_s390x.go@@ -377,6 +377,12 @@ type Flock_t struct {        Pid    int32 }+type F_cnvrt struct {+       Cvtcmd int32+       Pccsid int16+       Fccsid int16+}+ type Termios struct {        Cflag uint32        Iflag uint32

Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

        // Needed for z/OS so that the characters are not garbled if ptyp* is untagged        cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}        if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {                return nil, nil, err        }

There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------

Instead of creating FcntlFcnvrt, I created Fcntl and third argument is set to interface{} type. I can send you a delta if you want.

Frank, it is ok for you to do a PR against golang.org/x/sys, the usually practice is that you also add a test case to syscall_zos_test.go for the new functions you added. run `go run mksyscall_zos_s390x.go` on a 3.1 system to recreate zsyscall_zos_s390x.go. 

Yeah, I was wondering about that.  Since vimport didn't have it, I chose not to.  But you're probably right.

Frank, you should retrieve the tag on the file descriptor and check its value and text bit and only apply fcntl if it is really needed.

So I figured out the codepage issue.  I was searching through some of the z/OS Open Tools ports and found this in vimport:

diff --git a/src/pty.c b/src/pty.cindex f11a22dcd..b8dea655f 100644--- a/src/pty.c+++ b/src/pty.c@@ -194,6 +194,12 @@ mch_openpty(char **ttyn)     if ((f = posix_openpt(O_RDWR | O_NOCTTY | O_EXTRA)) == -1) 	return -1; +#ifdef __MVS__+    // Needed for z/OS so that the characters are not garbled if ptyp* is untagged+    struct f_cnvrt cvtreq = {SETCVTON, 0, 1047};+    fcntl(f, F_CONTROL_CVT, &cvtreq);+#endif+     // SIGCHLD set to SIG_DFL for grantpt() because it fork()s and     // exec()s pt_chmod     sigcld = mch_signal(SIGCHLD, SIG_DFL);

I first made some changes to golang.org/x/sys/unix:

diff --git a/unix/syscall_zos_s390x.go b/unix/syscall_zos_s390x.goindex 312ae6a..84fefae 100644--- a/unix/syscall_zos_s390x.go+++ b/unix/syscall_zos_s390x.go@@ -511,6 +511,7 @@ func Faccessat2(dirfd int, path string, mode uint32, flags int) (err error) { //sys  Fchown(fd int, uid int, gid int) (err error) //sys  Fchownat(fd int, path string, uid int, gid int, flags int) (err error) = SYS___FCHOWNAT_A //sys  FcntlInt(fd uintptr, cmd int, arg int) (retval int, err error) = SYS_FCNTL+//sys  FcntlFcnvrt(fd uintptr, cmd int, arg *F_cnvrt) (retval int, err error) = SYS_FCNTL //sys  Fdatasync(fd int) (err error) = SYS_FDATASYNC //sys  fstat(fd int, stat *Stat_LE_t) (err error) //sys  fstatat(dirfd int, path string, stat *Stat_LE_t, flags int) (err error) = SYS___FSTATAT_A@@ -728,6 +729,18 @@ func Ptsname(fd int) (name string, err error) {        return }--- a/unix/ztypes_zos_s390x.go+++ b/unix/ztypes_zos_s390x.go@@ -377,6 +377,12 @@ type Flock_t struct {        Pid    int32 }+type F_cnvrt struct {+       Cvtcmd int32+       Pccsid int16+       Fccsid int16+}+ type Termios struct {        Cflag uint32        Iflag uint32

Then in the new pty_zos.go file in  https://github.com/creack/pty I added the following right after the call to unix.Posix_openpt():

        // Needed for z/OS so that the characters are not garbled if ptyp* is untagged        cvtreq := unix.F_cnvrt{Cvtcmd: unix.SETCVTON, Pccsid: 0, Fccsid: 1047}        if _, err = unix.FcntlFcnvrt(uintptr(ptmxfd), unix.F_CONTROL_CVT, &cvtreq); err != nil {                return nil, nil, err        }

There remains one issue, however.  It's treating EBCDIC NL (x15) as just a linefeed.  That is, instead of placing the cursor at the beginning of the next line, it places at the same position as the end of the previous line, just down one line.

VIM seems to handle this correctly, so I'm sure it's fixable.  Just have to figure out how.

By the way, is it OK for me to do a PR against golang.org/x/sys?  Or is that something the z/OS Go team would want to do?

Yes, of course that is needed.  I'm just not sure where it belongs.  It seems like there should be a general solution, maybe within the Go runtime itself.  I don't know where it would go in the PTY package.  The PTY package includes only the following:

func Getsize(t *os.File) (rows, cols int, err error)func InheritSize(pty, tty *os.File) errorfunc Open() (pty, tty *os.File, err error)func Setsize(t *os.File, ws *Winsize) errorfunc Start(cmd *exec.Cmd) (*os.File, error)func StartWithAttrs(c *exec.Cmd, sz *Winsize, attrs *syscall.SysProcAttr) (*os.File, error)func StartWithSize(cmd *exec.Cmd, ws *Winsize) (*os.File, error)func GetsizeFull(t *os.File) (size *Winsize, err error)

It's not involved with the reads and writes themselves.

Hello Frank that's an interesting problem.
One solution that comes to mind is to use EBCDIC to ASCII conversion functions we have in ibmruntimes/go-recordio. 
Here's a codepage conversion example.
https://github.com/ibmruntimes/go-recordio/blob/main/v2/example-codepage/main.go

Please let me know your thoughts on using go-recordio as a solution.

------------------------------
Hyder Naqvi

Original Message:
Sent: Thu May 23, 2024 01:51 AM
From: Frank Swarbrick
Subject: Codepage conversion issue

So I've gotten the "micro" editor (https://github.com/zyedidia/micro) running well.  I've only found an issue with one nagging little thing.  There is a function that allows opening of a psuedo-terminal within a micro application window.  In order to make this work I had to make changes to various different programs/APIs.  The issue is that when you write to the tty and then read back from the pty, the original text is in UTF-8 and the text read from the pty is in EBCDIC.  I'm not sure how this may be resolved.  Any thoughts are welcome.

Here are the changes I made:

• In project micro:
  • Added zos build tag to internal/action/actions_posix.go and internal/action/terminal_supported.go and !zos tag to internal/action/terminal_unsupported.go.
  • In go.mod, changed version of github.com/mattn/go-isatty required from v0.0.11 to v0.0.13 (which already has z/OS support).
• In project https://github.com/zyedidia/terminal:
  • Added zos build tags to ioctl_posix.go and vt_posix.go.
• In https://github.com/zyedidia/tcell:
  • Created tscreen_zos.go from tscreen_solaris.go.
• In project https://github.com/creack/pty: I
  • Added the !zos build tag to pty_unsupported.go
  • Added new file pty_zos.go, which is as follows:

//go:build zos// +build zospackage ptyimport (        "golang.org/x/sys/unix"        "os"        "syscall")func open() (pty, tty *os.File, err error) {        ptmxfd, err := unix.Posix_openpt(unix.O_RDWR)        if err != nil {                return nil, nil, err        }        p := os.NewFile(uintptr(ptmxfd), "/dev/ptmx")        if p == nil {                return nil, nil, err        }        // In case of error after this point, make sure we close the ptmx fd.        defer func() {                if err != nil {                        _ = p.Close() // Best effort.                }        }()        sname, err := unix.Ptsname(ptmxfd)        if err != nil {                return nil, nil, err        }        _, err = unix.Grantpt(ptmxfd)        if err != nil {                return nil, nil, err        }        if _, err = unix.Unlockpt(ptmxfd); err != nil {                return nil, nil, err        }        ptsfd, err := syscall.Open(sname, os.O_RDWR|syscall.O_NOCTTY, 0)        if err != nil {                return nil, nil, err        }        t := os.NewFile(uintptr(ptsfd), sname)        if err != nil {                return nil, nil, err        }        return p, t, nil}

The only changes that really matter (as far as I can tell) are the ones to PTY.  If you run "go test" after making the changes above you should see the issue present itself (as well as one other that doesn't seem to matter much.)

------------------------------
Frank Swarbrick
------------------------------