Working With URLs Using Go’s net/url Package

URLs (Uniform Resource Locators) are one of the internet’s most critical pieces of infrastructure. As you build web applications, you’ll need to manipulate URLs to locate and retrieve resources.

When you build more sophisticated web apps, you’ll need to work with URLs at a finer-grained level. You might need to identify the scheme, hostname, path, and query parameters. You’ll also need to know how to encode and decode URLs so you can handle special characters and keep your web application safe.

Go’s standard library provides the net/url package to handle URLs and URL components.

The URL Package

The url package provides comprehensive functions and features for working with URLs and their separate parts. It provides functions for parsing, constructing, encoding, and decoding URLs, making the package useful for web development.

Some of the key features of the url package are the ability to parse URLs into individual components for manipulation and URL construction for HTTP requests. The url package also provides a URL struct with a Parse method for parsing strings into URLs.

Here’s the url.URL struct model:

        package main

type URL struct {
    // Scheme is the protocol scheme of the URL,
    // such as "http" or "https"
    Scheme string

    // Opaque is used to hold any opaque data
    // that should be encoded in the URL
    Opaque string

    // User holds information about the user making the request,
    // such as a username and password
    User *Userinfo

    // Host is the hostname or IP address
    // of the server hosting the URL
    Host string

    // Path is the path of the URL on the server
    Path string

    // RawPath is the original,
    // encoded path of the URL
    RawPath string

    // ForceQuery indicates whether the URL should include a query string
    // even if it is empty
    ForceQuery bool

    // RawQuery is the original,
    //encoded query string of the URL
    RawQuery string

    // Fragment is the fragment identifier of the URL, 
    // used for linking to a specific element on a page
    Fragment string

    // RawFragment is the original,
    // encoded fragment identifier of the URL
    RawFragment string
}

Knowing how to access various parts of the URL struct can be useful for tasks like validation.

Parsing URLs Using the Parse Function

The Parse function of the url package provides the functionality for parsing URL strings into Individual components. The Parse function takes a single URL as an argument and returns a pointer to the url.URL struct containing the parsed data of the URL and an error type.

Here’s how you can use the Parse function to retrieve the elements of a URL.

        import (
    "fmt"
    "net/url"
)

func main() {
    // The URL you want to parse
    exampleURL := "https://www.example.com/path?param1=value1&param2=value2"

    // Parse the URL
    parsedURL, err := url.Parse(exampleURL)

    if err != nil {
        fmt.Println(err)
        return
    }

    // Print all the fields of the URL
    fmt.Println("Scheme:", parsedURL.Scheme)
    fmt.Println("Opaque:", parsedURL.Opaque)
    fmt.Println("User:", parsedURL.User)
    fmt.Println("Host:", parsedURL.Host)
    fmt.Println("Path:", parsedURL.Path)
    fmt.Println("RawPath:", parsedURL.RawPath)
    fmt.Println("ForceQuery:", parsedURL.ForceQuery)
    fmt.Println("RawQuery:", parsedURL.RawQuery)
    fmt.Println("Fragment:", parsedURL.Fragment)
    fmt.Println("RawFragment:", parsedURL.RawFragment)
}

The exampleURL variable holds the full, unparsed URL, and the Parse function parses the content of the exampleURL variable and returns the parsed URL. The program ends with a series of calls to Println to demonstrate the individual fields of the URL struct.

result of the url.URL struct instance for the example URL

The parse function doesn’t check if a URL is a real one that actually exists, it only parses the URL syntactically. You can use the http package to make a GET request to the URL and check the response:

        import (
    "fmt"
    "net/http"
)

func main() {
    // The URL you want to check
    exampleURL := "https://www.example.com"

    // Make an HTTP GET request to the URL
    response, err := http.Get(exampleURL)

    if err != nil {
        fmt.Println(err)
        return
    }

    defer response.Body.Close()
    
    // Check the response status code
    if response.StatusCode == http.StatusOK {
        fmt.Println("URL exists.")
    } else {
        fmt.Println("URL does not exist.")
    }
}

The main function makes a GET request to the exampleURL using the Get function of the http package. That function returns a response instance and error type. The program ends with an if statement to validate the website’s existence by checking the HTTP status code against the StatusOk constant from the http package.

This approach allows you to take actions based on the outcome of the check, such as redirecting the user to a different page, displaying an error message, or retrying the request after a certain period.

Encoding and Decoding URL Parameters

The url package provides the Encode method for encoding URL parameters. The Encode function percent-encodes special characters and spaces in URL parameters.

        import (
    "fmt"
    "net/url"
)

func main() {
    // create a new url.Values struct
    params := url.Values{}

    // add values to the struct
    params.Add("name", "John Smith")
    params.Add("age", "30")
    params.Add("gender", "male")

    // encode the struct into a string
    encodedParams := params.Encode()
    fmt.Println(encodedParams)

    // Output: "age=30&gender=male&name=John+Smith"
}

The main function creates a new Values struct instance of the url package, and the Add method of the struct instance adds key-value pairs of data to the struct instance.

The Encode method converts the key-value pair to the URL string format "key1=value1&key2=value2&key3=value3".

You can decode an encoded URL string with the ParseQuery function of the url package.

        import (
    "fmt"
    "net/url"
)

func main() {
    // encoded string of URL parameters
    encodedParams := "age=30&gender=male&name=John+Smith"

    // parse the encoded string into a url.Values struct
    params, err := url.ParseQuery(encodedParams)

    if err != nil {
        fmt.Println(err)
    }

    // print the struct
    fmt.Println(params)

    // Output: map[age:[30] gender:[male] name:[John Smith]]
}

The encodedParameter variable is an encoded URL string. The ParseQuery function takes in the encodedParameter variable and returns the decoded URL string and an error.

These Go Packages Can Take Your Web Routing Game to the Next Level

The URLs you use for the pages of your web application contribute to its performance and visibility on search engines. Web routing is the process of directing incoming requests to the appropriate handler function based on the URL.

You can route using the http package or popular third-party packages like Gorilla Mux, Chi, Pat, or Httprouter. These packages make routing easier than the http package by abstracting away some of its complexities.