BurrowedInCode

Go HTTP Client: Configure Once, Reuse Often

Recently, I’ve been working on developing a CLI download manager in Go. Apart from the UI, it relies entirely on Go’s standard library. I’ll discuss the project in a future post, but for now, let’s focus on its HTTP client.

Go provides an excellent standard library; it includes everything we need to make HTTP requests. No third-party packages required. In this post, we’ll configure Go’s http.Client once and reuse it across requests, using http.NewRequestWithContext and Client.Do.

Sending HTTP Requests

If you come from the world of JavaScript you’re probably familiar with the fetch API. It’s easy to use; you pass in a URL, an optional request body, then you handle the response. Simple!

Go also provides an easy-to-use HTTP client but it gives you more control over how your requests are handled

Let’s start with the http.Client struct.

package main

import "net/http"

func main() {
client := &http.Client{}
_ = client
}

http.Client is a configurable HTTP client provided by Go’s standard library. That’s pretty much it!

Although it offers several configuration options, there are two that are worth discussing in this post: Timeout and Transport.

Timeout

The Timeout field specifies how long the client can take to make the request before cancelling it, as if the request’s context ended. Timeout covers the entire request lifecycle, including reading the response body.

client := &http.Client{
  Timeout: 2 * time.Second,
}

We tell the client here that a request should not last longer than two seconds, otherwise cancel it.

Here’s a simple example demonstrating how to configure and test an HTTP client timeout in Go.

package main

import (
	"fmt"
	"net/http"
	"time"
)

func main() {
	client := &http.Client{
		Timeout: 2 * time.Second,
	}

	url := "https://httpbin.org/delay/5"

	res, err := client.Get(url)
	if err != nil {
		fmt.Println("request failed: ", err)
		return
	}
	defer res.Body.Close()

  // Will not execute as the request times out
	fmt.Println(res.StatusCode) 

}

We configure the client with a two-second timeout and send a request to the API endpoint. The provided endpoint intentionally responds after five seconds. We build and send the request but due to the configured timeout we get this error message:

request failed: Get "https://httpbin.org/delay/5": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Try it on your own!

Transport

The Transport field is for configuring how the client communicates with the server. For example, you can configure connection pooling:

client := &http.Client{
    Transport: &http.Transport{
        MaxIdleConns:        10,
        IdleConnTimeout:     30 * time.Second,
        MaxIdleConnsPerHost: 5,
    },
}

Here is what each field does:

If you don’t need a custom configuration, you can use http.DefaultClient, a predefined HTTP client.

Reusing the client

Once we’ve configured an http.Client, we can reuse it across multiple requests.

Here I’d like to introduce two methods: http.NewRequestWithContext and client.Do. I’ll use these methods in the following example to demonstrate how we can reuse the same client to send multiple HTTP requests.

package main

import (
	"context"
	"fmt"
	"net/http"
	"time"
)

func main() {
	client := &http.Client{
		Timeout: 10 * time.Second,
	}

	urls := []string{
		"https://httpbin.org/get",
		"https://httpbin.org/headers",
	}

	for _, url := range urls {
		req, err := http.NewRequestWithContext(
			context.Background(),
			http.MethodGet,
			url,
			nil,
		)
		if err != nil {
			fmt.Println(err)
			continue
		}

		res, err := client.Do(req)
		if err != nil {
			fmt.Println(err)
			continue
		}

		fmt.Println(res.StatusCode)
		res.Body.Close()
	}
}

Inside main, we create an HTTP client with a configured timeout Timeout. We then declare a slice of URLs and iterate over them.

http.NewRequestWithContext is similar to http.NewRequest, except it allows us to associate a request with a specific context. I won’t go into detail about contexts in this post.

After handling any errors that occur while creating the request, we call client.Do. This method sends the request using our configured client and returns an HTTP response or an error.

By reusing the same HTTP client, we also reuse its underlying transport, which manages connection pooling. This approach allows you to use the same connection across requests in contrast to using a new connection for every request.

To learn more about the capabilities of net/http check out the Docs.