Go http middleware chain with context package

Middleware is a function which wrap http.Handler to do pre or post processing of the request.

Chain of middleware is popular pattern in handling http requests in go languge. Using a chain we can:

  • Log application requests
  • Rate limit requests
  • Set HTTP security headers
  • and more

Go context package help to setup communication between middleware handlers.

Implementation

Middleware chain

To compose handlers into a chain I choose Alice package which implement logic of looping through middleware functions.

chain := alice.New(security.SecureHTTP, httpRateLimiter.RateLimit, myLog.Log).Then(myAppHandler)    

http.ListenAndServe(":8080", chain) 

All source code is available at GitLab.

The only requirement of Alice that handlers should follow:

func (http.Handler) http.Handler

Example of SecureHTTP handler:

func (security *SecurityHTTPOptions) SecureHTTP(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		for header, value := range security.headers {
			w.Header().Add(header, value)
		}

		h.ServeHTTP(w, r)
	})
}

Context – communicator between middleware calls

Using go context package we can pass objects between middleware functions with:

  • context.WithValue
  • http.Request.WithContext
  • http.Request.Context().Value()
func (this *Logger) Log(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		ctx := context.WithValue(r.Context(), LoggerContextKey, this)
		t1 := time.Now()
                h.ServeHTTP(w, r.WithContext(ctx))
		t2 := time.Now()
		this.Printf("%s %q %v\n", r.Method, r.URL.String(), t2.Sub(t1))
	})
}

Where LoggerContextKey is a key by which we can use later to extract logger object:

func Recover(h http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		defer func() {
			if err := recover(); err != nil {
				mylog := r.Context().Value(logger.LoggerContextKey).(*logger.Logger)
				mylog.Printf("panic: %+v", err)
				http.Error(w, "500 - Internal Server Error", http.StatusInternalServerError)
			}
		}()

		h.ServeHTTP(w, r)
	})
}

Constructors

Sometimes you need to initialize objects before they are ready to use. To go around Alice limitation on func signature we can make a wrapper type for a particular package. Let see on log package example:

Wrapper type

type Logger struct {
	*log.Logger
}

Constructor

func InitLogger() *Logger {
	return &Logger{log.New(os.Stdout, "logger: ", log.Lmicroseconds)}
}

In the main program we use it like:

myLog := logger.InitLogger()

Conclusions

  • usually middleware functions have to be small and fast, so they don’t impact on overall application performance
  • chain is just a simple loop. You can write your own to support customized order of calls
  • create new custom type when you need to adjust other libraries to be useful in middleware pattern

You can find full demo of middleware chains at GitLab.

calculating md5 sum using go lang

Hi guys,
this is day 18 out of 100 days of code.

To get md5 sum of a file you crypto/md5 library and io/ioutil for reading files.

package main

import (
	"crypto/md5"
	"fmt"
	"io/ioutil"
	"log"
	"os"
)

func main() {
	if len(os.Args) <= 1 {
		log.Fatal("expected filename as parameter")
	}
	filename := os.Args[1]

	data, err := ioutil.ReadFile(filename)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("MD5 (%s) = %x\n", filename, md5.Sum(data))
}

Thanks!

Slow algorithm to insert strings into indexed file in alphabetical order

Hi guys,
this is day 17 out of 100 days of code in go lang.

Today I have played with files, sorting, bytes and strings. The goal was to create a kind of indexed file where all lines are arranged alphabetically.
What I got is very slow algorithm which took 15 seconds to insert 10K strings in an empty file and 49 seconds to insert another 10K strings to the same file. So, it slowdown very quickly as size grow.

func writeLineToFileSorted(newdata string) {
	indexData, err := ioutil.ReadFile("db.txt")
	if err != nil {
		log.Fatal(err)
	}
	var newIndexData []string
	for _, line := range bytes.Split(indexData, []byte("\n")) {
		//fmt.Printf("%s\n", line)
		newIndexData = append(newIndexData, string(line))
	}
	newIndexData = append(newIndexData, string(newdata))
	sort.Strings(newIndexData)

	bytesData := []byte(strings.Join(newIndexData, "\n"))
	ioutil.WriteFile("db.txt", bytesData, 0644)
}

I have to convert between strings and bytes which was kind of annoying, but maybe I am doing it wrong? Please let me know in comments.

If you want to load test it here is how:

func main() {
	rand.Seed(time.Now().UnixNano())
	var newdata string
	for i := 0; i < 10000; i++ {
		newdata = randStringRunes(10)
		writeLineToFileSorted(newdata)
		fmt.Println(i)
	}

}

var letterRunes = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ")

func randStringRunes(n int) string {
	b := make([]rune, n)
	for i := range b {
		b[i] = letterRunes[rand.Intn(len(letterRunes))]
	}
	return string(b)
}

Source code at Github

Happy coding!

Random image file download with go

Hi guys,
this is day 16 out of 100 days of code.

With help of http and ioutil packages file download and storage is quite easy task.

package main

import (
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	url := "https://picsum.photos/200/300/?random"

	resp, err := http.Get(url)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	image, err := ioutil.ReadAll(resp.Body)
	if err != nil {
		log.Fatal(err)
	}
	err = ioutil.WriteFile("./myimage.gif", image, 0644)
	if err != nil {
		log.Fatal(err)
	}
}

In the example I am using Lorem Picsum service to get random image every run.

Happy coding!

Github issues reader in go lang

Hi guys,
this is day 15 out of 100 days of go land coding.

I continue github subject and today it is github issue reader. Example demonstrate how API results which come in json format convert into struct type.

package main

import (
	"encoding/json"
	"flag"
	"fmt"
	"log"
	"net/http"
)

func main() {
	repo := flag.String("repo", "", "github owner/repo e.g. golang/go")
	id := flag.Int("id", -1, "issue id")
	flag.Parse()

	if *id == -1 || *repo == "" {
		log.Fatal("--repo and --id parameters must be provided")
	}
	issue, _ := read(*repo, *id)
	fmt.Print(issue.Title)
}

// IssueData - specify data fields for new github issue submission
type IssueData struct {
	Title string `json:"title"`
	Body  string `json:"body"`
}

func read(ownerRepo string, id int) (*IssueData, error) {
	apiURL := fmt.Sprintf("https://api.github.com/repos/%s/issues/%d", ownerRepo, id)
	resp, err := http.Get(apiURL)
	if err != nil {
		log.Fatal(err)
		return nil, err
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusOK {
		log.Fatal(err)
		return nil, err
	}

	var result *IssueData
	if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
		resp.Body.Close()
		return nil, err
	}

	return result, nil
}

Happy coding!

Create github issue ticket with golang

Hi guys,
today is day 14 out of 100 days of golang coding.

This time I was playing with github issues api and made small script which create new issues in given repository. However to use the code you would need to obtain github personal token key.

The interesting part was to make http post request with custom headers.

package main

import (
	"bytes"
	"encoding/json"
	"flag"
	"fmt"
	"io/ioutil"
	"log"
	"net/http"
)

func main() {
	token := flag.String("token", "", "github auth token")
	repo := flag.String("repo", "", "github owner/repo e.g. golang/go")
	title := flag.String("title", "", "title for new issue")
	body := flag.String("body", "", "body for new issue")
	flag.Parse()

	if *token == "" || *title == "" || *repo == "" {
		log.Fatal("--token, --repo and --title parameters must be provided")
	}
	create(*repo, *title, *body, *token)
}

// NewIssue - specify data fields for new github issue submission
type NewIssue struct {
	Title string `json:"title"`
	Body  string `json:"body"`
}

func create(ownerRepo, title, body, token string) {
	apiURL := "https://api.github.com/repos/" + ownerRepo + "/issues"
	//title is the only required field
	issueData := NewIssue{Title: title, Body: body}
	//make it json
	jsonData, _ := json.Marshal(issueData)
	//creating client to set custom headers for Authorization
	client := &http.Client{}
	req, _ := http.NewRequest("POST", apiURL, bytes.NewReader(jsonData))
	req.Header.Set("Authorization", "token "+token)
	resp, err := client.Do(req)
	if err != nil {
		log.Fatal(err)
	}
	defer resp.Body.Close()

	if resp.StatusCode != http.StatusCreated {
		fmt.Printf("Response code is is %d\n", resp.StatusCode)
		body, _ := ioutil.ReadAll(resp.Body)
		//print body as it may contain hints in case of errors
		fmt.Println(string(body))
		log.Fatal(err)
	}
}

Example usage is as follow:

go build github_issue.go
./github_issue --token="5daf49b235c41d53ba6fsfasdfasdfasfsad" --repo="vorozhko/go-tutor" --title="my new issue 3" --body="test"

Source code at Github.

 

github issue tracker with categories by date in go

Hi guys,
today is day 13 out of 100 days of code in go!

This time it is example of fetching issues from github and categorizing by created date.
To run example you need to install github issue searcher package from The Go Progamming Language book

go get gopl.io/ch4/github

Full code

package main

import (
	"fmt"
	"log"
	"os"
	"time"

	"gopl.io/ch4/github"
)

func main() {
	result, err := github.SearchIssues(os.Args[1:])
	if err != nil {
		log.Fatal(err)
	}
	fmt.Printf("%d issues:\n", result.TotalCount)
	var dateformat string
	for _, item := range result.Items {
		days := int(time.Since(item.CreatedAt).Hours()) / 24
		if days < 30 {
			dateformat = "less than a month old"
		} else if days < 365 {
			dateformat = "less than a year old"
		} else {
			dateformat = "more than a year old"
		}
		fmt.Printf("%s, #%-5d %9.9s %.55s\n", dateformat, item.Number, item.User.Login, item.Title)
	}
}

Happy coding!

Sort map by it’s keys in go

Hi guys,
this is day 12 out of 100 days of code. I didn’t post for a while, because I did a small break for hackaton, but now I am back on track.

When you print a map using range function it will access elements in randomized order. So, if you want to print map in sorted order you have to sort keys first. See how:

package main

import (
	"fmt"
	"sort"
)

func main() {
	m := map[string]int{"A": 1, "B": 2, "AA": 1, "C": 3, "BBB": 2}

	//reverse keys and values of m map
	var mapKeys []string
	fmt.Print("Unsorted:\n")
	for k, v := range m {
		fmt.Printf("%s -> %d\n", k, v)
		mapKeys = append(mapKeys, k)
	}

	sort.Strings(mapKeys)
	fmt.Print("Sorted:\n")
	for _, v := range mapKeys {
		fmt.Printf("%s -> %d\n", v, m[v])
	}
}

Solution is to use additional array for keys and sort it in wished order.

Happy coding!

how to sort map values in go

Hi guys,
this is day 11 out of 100 days of go coding.

This time it is very short demo of go maps and simple sort by map values. Note that the example below will not work correctly if values has duplicates.

package main

import (
	"fmt"
	"sort"
)

func main() {
        //init a map with strings keys and int values
	var m = make(map[string]int)
	m["Sun"] = 7
	m["Sat"] = 6
	m["Fri"] = 5
	m["Thu"] = 4
	m["Mon"] = 1
	m["Tue"] = 2
	m["Wed"] = 3

	//reverse keys and values of m map
	var days = make(map[int]string)
	//array of indexes for sorting
	var daykeys = make([]int, len(m))

	fmt.Print("Unsorted days of week\n")
	counter := 0
	for k, v := range m {
		fmt.Printf("%s -> %d\n", k, v)
		days[v] = k
		daykeys[counter] = v
		counter++
	}
	//sort indexes array here
	sort.Ints(daykeys)

	fmt.Print("Sorted days of week\n")
	for _, v := range daykeys {
		fmt.Printf("%s\n", days[v])
	}
}

I believe this code has a lot of room for optimization, so if you know how please write me in comments or ping me in twitter with your version.

Happy coding everyone!