Reflection in Go

I’m going to walk you through how to understand reflection in Go using the reflect package. We’ll do this by looking at an open-source package I created called go-flags that utilizes reflection heavily.

The use of reflection is often frowned upon because it side steps the ‘compile time’ type safety we get in Go. So I’ll also explain how we can still ensure a level of type safety even at runtime.

What is reflection?

Reflection in computing is the ability of a program to examine its own structure, particularly through types; it’s a form of metaprogramming. It’s also a great source of confusion.
– The Laws of Reflection

The above quote comes from the opening of the Go Blog post The Laws of Reflection and I strongly recommend you read it, as it breaks down how types and interfaces work in Go (which is a prerequisite to understanding reflection) before leading into how reflection itself works.

There will be a summarized version of that information here, but ultimately what differentiates their post from mine is that I’ll be discussing examples that are more practical and based on a real project (e.g. go-flags).

Understanding Interfaces in Go

In Go an ‘interface type’ is a collection of methods which can be defined like so:

type Foo interface {
	Bar(s string)
	Baz(i int)
}

If you define a variable to be of type Foo, then you’re saying the value that is assigned to the variable can be any type that implements the methods as defined by the Foo interface.

Conceptually the variable containing the assigned value can be thought of as a tuple that contains the underlying concrete value and the concrete type.

This is best demonstrated in an example. Below we create an interface type Foo, and then we define a struct type that will implement that interface. We’ll assign an instance of the struct to a variable defined as being an interface type of Foo and we’ll inspect the code at runtime using the reflect package to see what it can tell us about the variable and what it contains:

package main

import (
	"fmt"
	"reflect"
)

type Foo interface {
	Bar(s string)
	Baz(i int)
}

type T struct{
	s string
	i int
}

func (t T) Bar(s string) {
	//
}

func (t T) Baz(i int) {
	//
}

func main() {
	var f Foo
	f = T{"testing", 123}

	fmt.Printf("(%+v, %+v)\n", reflect.ValueOf(f), reflect.TypeOf(f))
}

The output of the above program is:

({s:testing i:123}, main.T)

OK, so we can see that the ‘value’ assigned to the f variable (of interface type Foo) is a struct with the relevant fields assigned with the appropriate value types. We can also see that the ‘type’ of the value is main.T, which is correct and expected.

So this clearly demonstrates that although we’ve stated the variable f should hold an interface type of Foo, it’s actually holding a different type entirely.

This probably sounds like common sense (and as such not that interesting) but if you really think about it, that’s incredible. We’ve not explicitly declared the T struct as implementing the Foo interface.

Instead the fact it does implement the interface, and doing so is being dynamically understood by the compiler is a super powerful feature.

Why do we need Reflection?

You need reflection whenever dealing with information that wasn’t known at the time of the program being written. One such example might be that you need to populate a struct with configuration data that comes from a external file.

There are many other examples, and the one I’ll be using today is similar in that I wanted to populate a struct with values provided by a user at runtime, specifically CLI flag values.

The building blocks of Reflection

So let’s consider what are probably the two primary reflection methods people learn about first when dealing with reflection (these were used in our example code earlier):

  • reflect.ValueOf(): returns a reflect.Value.
  • reflect.TypeOf(): returns a reflect.Type.

Both returned types have a bunch of methods associated with them that are unique to the purpose of those types. So for example, you want a reflect.Value if you’re interested in querying the concrete value, where as reflect.Type is more useful when you need to know information about the specific type.

So the use case for getting back a reflect.Type vs reflect.Value depends on what you’re trying to ascertain at runtime.

Once you have either a reflect.Value or a reflect.Type, then you can start digging further down into the data that is exposed via these types.

This is where exploring a real-world use case can be handy, as it helps to clarify why certain methods on these objects are called (compared to using a more contrived example).

Package Walk-through: Go-Flags

If you’ve ever written a CLI program in Go you’ll inevitably use its flag package. Most people find this package unintuitive and in some cases just downright difficult to work with or to build nice patterns of use from.

This leads people to build their own abstraction patterns on top of the standard library, but from my experience the majority (if not all) of third-party flag packages are convoluted and confusing to work with. The user experience feels very poor IMHO.

Note: I’m not knocking these packages. They all do very complex things, and solve real problems, and are written by much smarter people than myself. But it doesn’t change the fact that rarely do they come across as ‘simple’ and ‘easy to use’.

I wanted to solve the problem of handling flags and commands for a CLI based program in a simple way, that didn’t require me to construct a whole bunch of boilerplate code and have to negotiate lots of types.

In essence I wanted to define a ‘schema’ of what flags and commands (and flags for those commands) that I was expecting, and to have a package magically populate that struct with the values provided by a user of my program.

This is where utilizing reflection was the only real solution to my problem (if I indeed wanted the user experience to be as simple as possible).

So thus go-flags was born.

Here’s an example of how you would use it:

package main

import (
	"fmt"
	"os"

	"github.com/integralist/go-flags/flags"
)

type Schema struct {
	Debug   bool   `short:"d" usage:"enable debug level logs"`
	Number  int    `short:"n" usage:"a number field"`
	Message string `short:"m" usage:"a message field"`
	Foo     struct {
		AAA string `short:"a" usage:"does A"`
		BBB string `short:"b" usage:"does B"`
	}
	Bar struct {
		CCC bool `short:"c" usage:"does C"`
	}
}

func main() {
	var s Schema

	err := flags.Parse(&s)
	if err != nil {
		fmt.Printf("error parsing schema: %v\n", err)
		os.Exit(1)
	}
}

This is very simple and concise from a user’s perspective.

We can see that there are a bunch of top-level, non command specific, flags defined:

  • -d/-debug
  • -n/-number
  • -m/-message

You’ll notice that we utilize struct tags (the backticks that follow the struct field’s type) for defining the ‘short’ flag (while the field name itself is defining the ‘long’ flag variation) as well as defining the ‘usage’ description for each flag.

We can also see that we’ve defined two separate commands (Foo and Bar), and each command has its own set of flags:

  • foo: -a/-aaa (string), -b/-bbb (string).
  • bar: -c/-ccc (bool).

An example of how a user might then run this program would be:

my_cli_app -debug -n 123 -m "something here" foo -a beepboop -b 666

Explaining how go-flags uses Reflection

So how exactly does go-flags achieve this magic using reflection?

Let’s step through the code and find out…

Note: I’m not an expert on the reflect package, nor was a lot of time spent on this package outside getting it functional, so there’s likely many improvements that can be made to the code.

We’ll start with the Parse() function. Here we can see that the function accepts an argument of type interface{} which is the empty interface.

The reason for this is because we want the schema for the flags to be defined by the consumer of this package, and so at runtime we have no idea what that value will look like (hence the use of interface{} to mean: we’ll take any type).

Once we’ve got this unknown value we’ll execute the following code to get at the actual value:

v := reflect.Indirect(reflect.ValueOf(s))

We can see the familiar reflect.ValueOf() but we have also wrapped it in a call to reflect.Indirect(), so why do that? Well, the Indirect() method will dereference the struct pointer to the struct concrete value!

It’s also safe to use Indirect() because if you give it a non-pointer value it’ll just return the value from reflect.ValueOf(). Don’t worry, in just a moment we’ll do some validation of the given argument to ensure the user has followed our package instructions and indeed given us a pointer to a struct containing their flag schema.

Next we acquire a reflect.Type type from the given argument:

st := v.Type()

We get the type structure from the reflect.Value type returned from calling reflect.Indirect(), and we do this because later on we’ll pass it into a function called IterFields which iterates over the user’s schema struct. We pass it into that function because within there we’ll try to get the struct’s individual fields as a reflect.StructField (I’ll explain why we want that later).

Now it’s at this point we do some runtime type validation:

if v.Kind() != reflect.Struct {
  return ErrWrongType
}

We’re expecting a struct and there’s no point trying to go any further with something that’s not a struct, so we short-circuit the program if we’re given anything unexpected.

OK, at this point of our package we have a few stages that involve, among many things, iterating the given struct (we use a custom iterator function called IterFields and we pass it a callback function to execute for each ‘field’ in the given struct).

Here are the stages at a high-level:

  1. iterate over the top level fields of the user provided struct, and dynamically create flags.
  2. parse the flag values the user of the CLI program have provided (using the flags our package has now dynamically generated).
  3. iterate over the top level fields of the user provided struct, and populate the fields with the parsed flag values.
  4. identify the command the user provided when running the CLI program (e.g. foo or bar).
  5. parse the flag values the user provided after specifying the command.
  6. dynamically create a flag.FlagSet for the command.
  7. parse the flagset values the user provided (using the flagset our package has now dynamically generated).
  8. iterate over the command fields of the user provided struct, and populate the fields with the parsed flagset values.

Now we understand the general steps taken, we can dig into each of those and understand what reflection work needs to be carried out.

I won’t be discussing the general code logic, just picking out the bits related to reflection. So if you want to understand the full go-flags implementation, then I recommend you read through the code base in its entirety after finishing this post.

Note: if I was working with Go 2.0 then I’d have access to generics and parts of this code could be reduced/simplified. Hello future readers who are lucky enough to have Go 2.0.

As the majority of these steps are related to iterating over the user provided struct, let’s start with the IterFields function that enables that.

First thing we do in IterFields() is we call .NumField() on the reflect.Value, which gives us back the number of fields the struct contains.

This will panic if the reflect.Kind() is not a reflect.Struct type, hence why at the beginning of our Parse() function we do that validation there.

We could have moved the check for a struct down into the IterFields() function but then we’d end up doing extra unnecessary logic processing only to have to just stop the program any way (so best to short circuit the code as soon as possible).

Once we have the number of struct fields, we’ll create a loop for that number and we’ll use the incrementing i value to access each individual struct field by calling .Field() on the v variable’s reflect.Value type.

Note: you’ll find that when you call methods on a reflect.Value type you’ll likely end up with …another reflect.Value type!

for i := 0; i < v.NumField(); i++ {
  field := v.Field(i)

  ...
}

Next we call the .Field() method again but this time on the reflect.Type type. The reason we do this is because (as I mentioned earlier) we want a reflect.StructField type (which is what the .Field() call will give us) so we can inspect it later on in the callback function and parse out the ‘tags’ assigned to the current struct field.

We have at this point a if/else condition which I’m going to skip over the if block briefly and move onto discussing the else block which states:

if !recurse && field.CanSet() {
	callback(field, sf)
}

You’ll see we call .CanSet() on the field variable which (as noted earlier) is actually a reflect.Value type. The reason we do this is because one of the given callback functions will attempt to set a value onto the struct field and that is only valid if the struct field is public (i.e. exported) otherwise it would panic.

Let’s jump back up to the if statement, in there we’re checking the condition:

if field.Kind() == reflect.Struct {
  ...
}

This is a similar check to what we did at the start of the Parse() function, so why are we doing it again? Well, this time (as we iterate over the user’s schema struct and look at each field) we might discover one of the fields is itself a struct! So we want to also iterate over the nested struct’s fields looking for flag information.

All the code within this if block is effectively the same as what came before it with regards to getting access to the underlying struct ‘value’ that is assigned to the current struct’s field, and then looping over that nested structs own fields.

We could have done this with recursion but then it makes the IterFields() function more complex and as I was designing go-flags to work with only one level of nesting supported it wasn’t worth the effort to implementing a strict recursive function.

That said, there is one aspect of the if block we should look at which is the nested if block. The nested if block is what actually does the looping over the nested struct’s fields, and it does this by first getting at the nested struct concrete value (as I mentioned a moment ago).

To get at that struct value we do:

reflect.TypeOf(field.Interface())

OK, so this needs some explaining! Firstly, we call .Interface() on the field variable (which is a reflect.Value type) and then we pass that ‘result’ to reflect.TypeOf().

The .Interface() method will get the underlying concrete value as an empty interface (i.e. interface{}) type, and so that’s what we pass to reflect.TypeOf().

The reason we use .Interface() first is because if we had just passed the field variable into reflect.TypeOf(), then we would have gotten reflect.Value as the type (and that’s no use to us! we want the struct type).

So why did we need the struct type? Well, after we start iterating over the nested struct’s fields, we again need to get a reflect.StructField so that when we execute the given callback function it’ll be able to parse any ‘tags’ found on the nested struct fields.

At this point in time we’ve explained some key bits as far as looping safely over the user’s struct, so let’s go back to what we do in some of the ‘callback’ functions that get executed for each struct field.

One thing we do is access the struct field’s tags, and we do this by calling .Tag.Get() on the reflect.StructField type. So in go-flags we ask users to add struct tags like so:

short:"..." usage:"..."

That’s two tags ‘short’ and ‘usage’. So if I want the ‘short’ tag I’d call .Tag.Get("short") and if I want the ‘usage’ tag I’d call .Tag.Get("usage"). Nice and simple.

When it comes to setting the field values we have to check the kind of the field (remember this is a reflect.Value type). The following switch statement demonstrates this:

switch field.Kind() {
case reflect.Bool:
	if b, ok := getter.Get().(bool); ok {
		field.Set(reflect.ValueOf(b))
	}
case reflect.Int:
	if i, ok := getter.Get().(int); ok {
		field.Set(reflect.ValueOf(i))
	}
case reflect.String:
	if s, ok := getter.Get().(string); ok {
		field.Set(reflect.ValueOf(s))
	}
}

The call to .Kind() returns a reflect.Kind type, which is an integer iota that maps constants to a numerical value for easier comparison. Hence each case is a check against some of those constants like reflect.Bool, reflect.Int and reflect.String.

Once we know what type the field is we type assert the flag value to the relevant type and then call .Set() on the field. You’ll notice that we can’t just pass the value into the .Set() method as we need to provide an argument of type reflect.Value.

To fix that we call reflect.ValueOf() and pass it the value we’re trying to set, and then pass that resulting reflect.Value type into .Set().

Conclusion

Well, that’s it! There is sooooo much more to reflection than what I have had time to describe in this post, but hopefully this has been enough to help you in the future if you ever stumble across a need for reflection.

One thing I would say before I go is: avoid reflection wherever possible. It’s a nightmare to work with and although we can code defensively and get back some runtime safety, that’s still no comparison to compile time safety. There’s also a performance cost to runtime reflection that can’t be ignored.

So do please think twice before reaching for reflection.