Go - Consul Get KV

Overview

Recently I was requested to connect to an instance of Consul to grab some information for a project. I have never heard of Consul before. There is many things that it can do and has built right into it. One of the features it has is a “Key / Value Store”. An organization or person can spin up an instance of Consul and store some basic information in it. Here I am learning and sharing.

Counsul’s Key / Value Store can be useful for a variety of uses, such as creditials that may change, you don’t want to hardcode, or leave in a text file on a server. This is also great for configuration and settings that the program may use.

Having a simple key-value store is great, especially where you don’t have to rely on a big SQL server (by comparison). Putting information like that in a SQL server also usually means there is a certain structure that is expected everything has to be in. For nearly a decade I would do this with a SQL server and came up with my own personal conventions and preferences, and there was never a “great” way. The way I did it was “great” because it worked and I knew the ins & outs, but it was not that flexible, and it often changed based on the project, product, department, standards, code base, frameworks, process flow, architecture, time, and company. Changing conventions based on necessity is not “great”, especially since all those necessities are inevitable, and isn’t really a convention. With the Consul KV Store there is no specified rigid format or rigid structure. There is a convention that they use and follow, which is very basic and yet flexible. The value stored can be a string, JSON, HCL, or YAML or really just about anything since the default is JSON.
One of the best things in my humble opinion is that Consul has a few ways of accessing the information including a built-in API. The API is RESTful, and does include endpoints for GET, PUT, DELETE, and more. The API also has authentication and can be sectioned by key or folder and more. I will only be using the GET and not using any authentication token.

In this I just want to go over quickly what I have learned trying to access some information from Consul. I will be using Postman in some examples, but use your preference for making an API request. I will then be using Go and the library provided by Hashicorp (the company that makes Consul) to access the KV Store.

Documentation

As always, gotta start with the documenation.

Consul has some great tutorials online and great documentation online with some great videos. I am going to link to some below.
https://learn.hashicorp.com/consul

Consul API Documentation
https://www.consul.io/api-docs/
https://www.consul.io/api-docs/kv
https://www.consul.io/api-docs#results-filtered-by-acls
https://www.consul.io/api-docs/kv

https://pkg.go.dev/github.com/hashicorp/consul/api?utm_source=godoc
https://github.com/hashicorp/consul
https://pkg.go.dev/github.com/hashicorp/consul/api#KV.Get

This is my “private” repo for the examples of this blog post.
https://gitlab.com/Acutis/blog-example-go-consul-get.git

Setup

For anonimity names and such are changed around. The Consul instance I will be showing will be accessing “http://dev.mycompany.com/". The standard port for Consul is 8500.

In theory, you could see the Key / Value store at “http://dev.mycompany.com:8500/ui/server1/kv/".

On this instance in the KV Store I made a “folder” for my tests, and then put in a key value. The “folders” are created when you create a new key value and then add a “/” to it. If you wanted a folder for “tests” it would be created as “tests/password”, and that would create a folder named “tests” and a key value of “passwords” that you can store some information in it. In this example I will be using the folder “tests”.

The key I created I named “get-me”. I did not know what I was doing, and wasn’t sure what to name it. Something to note, the value of this is not the key-value pair. Meaning inside the “get-me” is not necessarily a key-value pair telling me more information. The key is “get-me” and the value is the content that you put in (default to JSON). This helps with naming and folders and is just something to think about in the future.

The value of “get-me” initially I set at {"pie":"strawberry rhubarb"}. I was not fully aware of how this was going to work and what to put in.

You will see through some of the experiments what is happening and what is returned. I encourage anyone else to go through some simple exercises as well to see what I mean and then decide for yourself what the best organization is for your purposes and the best storage. For me, the eventual goal would be to store a configuration in JSON and have that parsed out / unmarshalled in the code to be used.

API

First, I wanted to see what the API endpoint was returning. I could not find on the server or the docs what to setup for a client endpoint, and that was more on me. I understably probably didn’t understand all the verbiage used and probably didn’t retain what should be pertinent, so I had to throw some darts blindfolded. Fortunately, it did not take long.

The server I was using was named http://dev.mycompany.com:8500/ and the key, “get-me” was at http://dev.mycompany.com:8500/ui/kv/tests/get-me. Being observant I could see that this was supposed to be for the “UI” only, which is why its in the URL. I threw that into Postman anyway and I surprising did get a response of the HTML.

Doing an inspection of the web page and going to the network tab I saw that the UI was using an API endpoint of http://dev.mycompany.com:8500/v1/kv/tests/get-me. That is very convienent. To confirm this, I also did a quick search and found some other people using examples that were the same structure as mine.

Plugging the proper API endpoint into Postman I got a response of…

[
    {
        "LockIndex": 0,
        "Key": "tests/get-me",
        "Flags": 0,
        "Value": "eyJwaWUiOiJzdHJhd2JlcnJ5IHJodWJhcmIifQ==",
        "CreateIndex": 9,
        "ModifyIndex": 9
    }
]

A response, awesome. BUT, you will notice that the “value” is encoded, and this is not exactly what I was expecting. I was expecting just the value that I put into the website and nothing else. This is in the documenation, as well as what all the fields mean. We even see the “Key” field, which is in the API request so this is some nice validation. The main field that we are worried about is the “Value” field, which according to the documentation is Base64 encoded. Pluggint the value into an only base 64 decoder I do see the text is what I entered, {"pie": "strawberry rhubarb"}.
Knowing this, I confirmed the results using a curl command on my terminal, and I was able to get the same results.

At this point, I did try a few different variations of the URL to see what else, and for the most part I did get 404 errors that the Key-Value was not there. I did the same thing in the Go code later on, and we will cover that.

I did not have to put in any token or authentication for this example. I just used the most basic of options, and made a simple GET request. From what I understand, there could be a token needed.

Go Request

Now to use this example and put it into some Go code. I noticed that Hashicorp has some Go libraries, links above. The documentation is a little bit less than what I am use to with some of the standard go libraries, but again, I think it was just me and I needed to get acclimated.

Thinking of the postman request, I realize I probably could just do a simple http/net request in Go and get it, but lets try this Hashicorp library.

The general thought that is that I would need to (1) initialize any interfaces and make the request, (2) parse and decode the value from the request, (3) unmarshal the json string to a golang struct.
The library for go is "github.com/hashicorp/consul/api", and I renamed this to “consulapi” as you will see. Looking through the documentation I noticed that there is a Client, so lets begin creating that.

package main

import (
	// dependencies
	"encoding/json"
	"fmt"

	// dependencies
	consulapi "github.com/hashicorp/consul/api"
)

func main() {
	// create consul client
	client, err := consulapi.NewClient(consulapi.DefaultConfig())
	if err != nil {
		panic(err)
	}
}

The consulapi.DefaultConfig() generates a new client with default values, and the NewClient() needed a config to be passed to it. With the default config there is no server address setup, so it won’t really do much. Let’s take that out of there and make a variable with what we needed.

Looking in the docs at the Config struct I see that there is a field for “Address”, which is what we want to change.
I guess we could either do…

	consulCfg := consulapi.DefaultConfig()
	consulCfg.Address = "http://dev.mycompany.com:8500"

…or..

	consulCfg := &consulapi.Config{
		Address: "http://dev.mycompany.com:8500",
	}

This will now connect to the server. Something I learned through experimentation, though its not explictly noted in the documenation (maybe I missed it), is the server address should just be the root. Based on the URL I had before, I tried http://dev.mycompany.com:8500/v1/kv/tests, http://dev.mycompany.com:8500/v1/kv, and http://dev.mycompany.com:8500/v1 all with and without the trailing slash. None of those worked, which may be a stumbling block for someone else as well and I hope this helps.

Continuing on, I noticed in the documentation that the client has a KV Store procedure to access the KV store, named just KV(). This returns a connection to the Key-Value store. If the address above is not correct, there will be some errors here.

Now we hae this for code…

package main

import (
	// dependencies
	"encoding/json"
	"fmt"

	// dependencies
	consulapi "github.com/hashicorp/consul/api"
)

func main() {
	consulCfg := &consulapi.Config{
		Address: "http://dev.mycompany.com:8500",
	}

	// create consul client
	client, err := consulapi.NewClient(consulCfg)
	if err != nil {
		panic(err)
	}

	// connect to a kv store
	kv := client.KV()
}

This won’t compile yet…but next we need to use that variable kv that is the key-value store and do something with it. As luck would have it, that connection has a Get() method that accepts a string for the key and query options. I don’t think we need any query options, and I have not looked at it. The string for the “key” though I set just as I mentioned before as “tests/get-me”. Similarly with the server address in the client, I tried to change this “key” around as well. It was bugging me a bit because I didn’t read and understand things that I saw an API call and then the address plus this key was missing the “v1/kv” in the middle of it all. When this library makes the connection it also does the detection to figure things like that out.

Looking at the code again this is what I had at this point…

package main

import (
	// dependencies
	"encoding/json"
	"fmt"

	// dependencies
	consulapi "github.com/hashicorp/consul/api"
)

func main() {
	consulCfg := &consulapi.Config{
		Address: "http://dev.mycompany.com:8500",
	}

	// create consul client
	client, err := consulapi.NewClient(consulCfg)
	if err != nil {
		panic(err)
	}

	// connect to a kv store
	kv := client.KV()

	pair, _, err := kv.Get("tests/get-me", nil)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Pair: %v\n", pair)
}

At this point I was able to get a response, and it was similar to the API response in Postman.

Pair: &{tests/get-me 9 9 0 0 [123 34 112 105 101 34 58 34 97 112 112 108 101 34 125]   }

One of the differences that I see with this output is that the array at the end seems to be a byte array or []byte data in go. Changint that fmt.Printf() to accept string(pair.Value) and seeing if anything breaks (which it didn’t) I am seeing in clear text the JSON string I originally entered, {"pie":"strawberry rhubarb"}. I was expecting that this was going to be base64 encoded like the Postman response, but it appears that the Go library is already doing that for me. Making my job so much easier.

Here, all I needed to do was unmarshal that JSON into a Go struct, which should be rote by now. Below is everything strung together.

package main

import (
	// dependencies
	"encoding/json"
	"fmt"

	// dependencies
	consulapi "github.com/hashicorp/consul/api"
)

func main() {
	// consulCfg := consulapi.DefaultConfig()
	// consulCfg.Address = "http://dev.mycompany.com:8500"

	consulCfg := &consulapi.Config{
		Address: "http://dev.mycompany.com:8500",
	}

	fmt.Printf("Addy: %v\n", consulCfg.Address)

	// create consul client
	client, err := consulapi.NewClient(consulCfg)
	if err != nil {
		panic(err)
	}

	// connect to a kv store
	kv := client.KV()

	pair, _, err := kv.Get("tests/get-me", nil)
	if err != nil {
		panic(err)
	}

	fmt.Printf("Pair: %v\n %v\n %v\n", pair, string(pair.Value), pair.Value)

	// NOTE: Not needed - when using the library things are decoded already.  Was expecting to use this.
	// decodeText, err := base64.StdEncoding.DecodeString(string(pair.Value))
	// if err != nil {
	// 	panic(err)
	// }
	//
	// fmt.Printf("Pair Decode Val: %v\n", decodeText)

	type response struct {
		Pie string `json:"pie"`
	}

	res := response{}
	json.Unmarshal(pair.Value, &res)
	fmt.Printf("Pair Json: %v\n", res)
	fmt.Printf("Pair Json Pie: %v\n", res.Pie)

	fmt.Println("-------- END --------")
}

Conclusion

Consul is a powerful and great tool to use. The API is great and easy to use. The endpoints are simple enough with tons of ability. Using the Go libraries are equally easy, with just a few things to remember.
Questions, comments, additional thoughts? Let me know in the comments.