Cgo: A Brief Review & Introduction to Callbacks

The ability to use cgo callbacks is now available in Go for z/OS v1.22. In this blog, we will review some Cgo basics as well as cover the new functionality that has been added.

What is Cgo?

First, as a piece of context, let's look at regular cgo. "Cgo" is a set of language utilities that allow for interop between Go and C. It allows us to call into any C code during the execution of a Go program as well as access definitions and constants. This comes in handy when we need to write a modern wrapper for a legacy C library in Go. (As many popular go modules do...)

In a Go source file, we can import the "C" pseudo-package below a comment to tell the Go compiler that it should be parsed as valid C code rather than a Go comment. We say that the "C" package is a pseudo-package because it doesn't exist within the compiler. It is generated at compile time if the Go compiler detects any source files that include it.

A simple Cgo example might look something like this:

// File: main.go
package main

/*
#include<stdio.h>

void MyCFunc() {
    printf("Hello from C!\n");
}
*/
import "C"
import "fmt"

func main() {
    fmt.Println("Hello from Go!")
    C.MyCFunc()
}

In this example, we define MyCFunc in a comment at the top of a Go source file. We can then access any of the identifiers in the C code by using the "C" prefix just like any other package. So when we run this program we expect to see the string "Hello from Go!" followed by "Hello from C!".

Through the "C" prefix we are also able to access more than just function calls. We can reference pre-processor definitions, file-scope constants, global variables, and even type definitions. (Do note that C types are separate from Go types, so conversion will be needed)

If you are extra curious, you can run the command go tool cgo on any Go source file that contains a Cgo import to generate the intermediary files that make all of the magic happen.

For more information on Cgo, check out the original announcement blog as well as the cgo command documentation.

What are Cgo Callbacks?

Calling into C from Go is exciting, but wouldn't calling back into Go from C be even more exciting? Luckily Go happens to have that exact feature! Cgo Callbacks. This lets us jump between Go and C as many times as we like. (i.e. Go calls C, that calls Go, that calls C, etc.). Behind the scenes, Cgo does all the work to ensure that the Go runtime is still operating just as it was before we left Go originally. This means that within a callback we have access to any regular feature that we could expect in Go.

Here is the example from above, but now with an additional callback step:

// File: main.go
package main

/*
#include"my_c_impl.h"
*/
import "C"
import "fmt"

//export MyGoCallbackFunc
func MyGoCallbackFunc() {
    fmt.Println("Hello from Go again!")
}

func main() {
    fmt.Println("Hello from Go!")
    C.MyCFunc()
}

// File: my_c_impl.h
void MyCFunc();

// File: my_c_impl.c
#include<stdio.h>
#include"_cgo_export.h"

void MyCFunc() {
    printf("Hello from C!\n");
    MyGoCallbackFunc();
}

This example will output the same text as the previous example followed by "Hello from go again!".

What is going on here? Things have suddenly gotten a lot more complicated...

The first thing to note is that when using callbacks, C preambles and headers cannot contain definitions, only declarations (No function bodies). This means that we now need to separate our program into 3 different files. Let's go over each one.

my_c_impl.h is fairly easy to understand. It just contains the declaration of MyCFunc() for visibility.

Its implementation file; my_c_impl.c has the definition for this function along with the actual callback being performed. In the context of C, this looks like any other function call. We just haven't seen the definition anywhere yet. This function will be made available to C by the _cgo_export.h header file, which is generated during compile time.

Finally in main.go we find the missing pieces. To have access to the C code we have written, my_c_impl.h is included in the preamble. Our new callback function MyGoCallbackFunc() exists here with the comment //export MyGoCallbackFunc placed directly above it. This comment tells the Go compiler that this function needs to be included in _cgo_export.h when it is generated, thus making it available to C as a callback.

That's it! We have now completed a Cgo callback. Of course, this is a very simple example. We haven't shown it here yet, but these functions can take parameters and return values just like any other call. You just need to be mindful of the type conversion between Go and C.

Note:

Another option we have is to ditch the header file, and manually tell our C code about this callback function using extern void MyGoFunc(); instead of the _cgo_export.h include. (This is essentially what is already happening with the generated include. Just manual). You can also remove my_c_impl.h if you manually declare each C function in the Go preamble. i.e. extern void MyCFunc();

Another Example

For our last trick, let's take a look at a more involved example.

Regular C programmers will be familiar with the qsort function (The std library sort implementation). As arguments, qsort takes the array to be sorted as well as a comparator function pointer that will be used to make ordering decisions under the hood. If we perform this qsort operation in a regular Cgo call, we could use a callback to a Go function as the comparator!

Let's see what that would look like. This example contains both a C and Go implementation of the comparator function:

// File: main.go
package main

/*
extern void DoSort();
*/
import "C"
import "unsafe"

//export goComp
func goComp(a, b unsafe.Pointer) int {
    aInt := *((*int32)(a))
    bInt := *((*int32)(b))
    if aInt > bInt {
        return 1
    } else if aInt < bInt {
        return -1
    }

    return 0
}

func main() {
    println("Starting Cgo Call...")
    C.DoSort()
}

// File: sort.c
#include <stdio.h>
#include <stdlib.h>

extern int goComp(const void*, const void*);

int comp (const void * elem1, const void * elem2)
{
    int f = *((int*)elem1);
    int s = *((int*)elem2);
    if (f > s) return  1;
    if (f < s) return -1;
    return 0;
}

void DoSort(int argc, char* argv[])
{
    int x[] = {4,5,2,3,1,0,9,8,6,7};
    int y[] = {4,5,2,3,1,0,9,8,6,7};

    qsort (x, sizeof(x)/sizeof(*x), sizeof(*x), comp);
    qsort (y, sizeof(y)/sizeof(*y), sizeof(*y), goComp);

    printf("C Comparison:\n");
    for (int i = 0 ; i < 10 ; i++)
        printf ("%d ", x[i]);
    printf("\n");

    printf("Go Comparison:\n");
    for (int i = 0 ; i < 10 ; i++)
        printf ("%d ", y[i]);
    printf("\n");
}

When we run this code, we will see the following output:

Starting Cgo Call...
C Comparison:
0 1 2 3 4 5 6 7 8 9
Go Comparison:
0 1 2 3 4 5 6 7 8 9

A couple of things to note here... I have gone the route of manually declaring the cgo functions instead of using the header file approach in the previous example. It's easy to do here because there is only one C function and one Go export, but the header method is preferred for larger development projects. We can also see one of the mappings between Go and C types. void* in C is represented as unsafe.Pointer in Go. Types don't always have a 1-1 mapping like this, int for example is a C.int in Go and will require a Go int() cast to use.

Otherwise, passing our exported goComp comparator as a parameter to qsort works just like we would expect. The performance hit that you would pick up from crossing between languages so much would make this a terrible idea for production level code, but it's still a cool example of ways we can use cgo callbacks.