Transform JSON and Make It Readable With jq

JavaScript Object Notation (JSON) is one of the world’s most popular data formats. It has widespread support and a simple specification. It’s easy to work with in many programming languages, especially those targeting web development.

But manual inspection of JSON data remains awkward. Although it is a human-readable text format, a JSON dataset might contain huge amounts of data. Sources won’t always format JSON in an easy-to-read form.

javaThe jq tool allows users to format, filter, and transform JSON data.

What Is jq?

Since it’s a command-line tool, you will generally use jq by typing commands into a terminal. There is also an excellent online playground available, which we cover in more detail below.

Normal operation revolves around filters and the application of a filter to some input JSON. You might use jq to fetch a single item from a set of many. Or you might remove certain fields from every item in a set, simplifying the data. You can even perform complex operations to translate the input into a different form.

How to Download and Install jq

The jq program has no external dependencies, meaning that it’s very easy to get started. Begin by downloading an executable binary for Linux, macOS, or Windows via the button on the jq home page. Once you have the program downloaded, you can run it straight from the command line. You might want to rename it (mv jq-osx-amd64 jq) for convenience, and you might need to make it executable (chmod +x jq).

Confirm that you can run jq by executing it from the command line with no arguments:

        $ ./jq

You should see some general usage information, beginning with a simple one-line summary such as the following:

        jq - commandline JSON processor [version 1.6]

If you struggle with the above approach, there are alternatives. The jq software has support for common package managers, and you can always experiment with the online sandbox in the meantime.

Basic jq Usage

The standard usage is:

        jq [options] <jq filter> [file...]

So, for example:

        $ jq '.' data.json

You can also pipe input in via another command like so:

        $ echo '{"name":"john"}' | jq '.'
{
  "name": "john"
}

This is most useful when, for example, the first command is something like a call to curl which can fetch JSON data from a web service.

The filter shown in these examples is the simplest possible, . (a period), which prints the input in a prettified form. This is already quite useful, but jq’s filters provide much more power than this.

How to Apply Basic Filters to JSON Using jq

A jq filter is a little like a CSS selector or an XPATH expression. It’s one long expression consisting of smaller parts. A complete filter might look complicated, but once you’ve learned the basics, each part should be understandable.

Working With Objects

You can fetch the value of an object property using the .property syntax:

        $ echo '{"name":"john"}' | jq '.name'
"john"

This can chain to access deep-nested structures:

        $ echo '{"name":{"first":"john","last":"smith"}}' | jq '.name.last'
"smith"

Working With Arrays

The simplest array operation returns one element via its index:

        $ echo '[1,2,3]' | jq '.[2]'
3

Note that, as with most programming languages, jq indexes arrays starting from position 0. You can also slice a subarray using this syntax:

        $ echo '[1,2,3]' | jq '.[1:3]'
[
  2,
  3
]

Without an index inside the square brackets, jq transforms a single array value into its own contents, as multiple values:

        $ echo '[1,2,3]' | jq '.[]'
1
2
3

This is an important method for chaining filters together, which we’ll show later.

More Advanced Features

You can only gain a full understanding of jq’s power by reading the jq manual. In fact, jq’s support for operators, variables, and even user-defined functions makes it capable of acting like any programming language.

These features make advanced usage possible, albeit complicated. But jq has some built-in features, such as functions and operators, that benefit even simple tasks. Here’s an example:

        $ echo '[2,4,8]' | jq 'add / length'
4.666666666666667

This filter feeds the input into both the add and length functions, dividing the results. In operation, it calculates the average of an array of numbers.

The division operator can also act on strings to split them based on a separator:

        $ echo '"Just testing"' | jq '. / " "'
[
  "Just",
  "testing"
]

The select function filters an array, keeping only those items that pass a given constraint:

        $ echo '[2,4,8]' | jq '.[] | select(. >= 3)'
4
8

Note that this is also an example of jq’s pipe operator (|) which is like a shell’s pipe. It feeds the result of its left-hand filter in as the input to its right-hand filter.

The map function is very useful when working with arrays. It carries out an operation on each element of the array rather than the whole array itself:

        $ echo '[1,2,3]' | jq 'map(.+1)'
[
  2,
  3,
  4
]

You will often use it in conjunction with select, e.g.

        $ echo '[2,4,8]' | jq 'map(select(. >= 3))'
[
  4,
  8
]

Putting It All Together: a Practical jq Example

Since jq processes any valid JSON piped to it, you can send it output from the curl command. This allows you to fetch JSON from a URL and immediately process it on the command-line:

JSON Feed is a JSON alternative to the RSS and Atom formats. The NPR site is one example that supports JSON Feed, but it’s difficult to view from source and contains a lot of data:

Straight away, you can see how much easier it is to read by fetching this data and piping it through jq:

        $ curl -s https://feeds.npr.org/1019/feed.json | jq '.'

Here’s a more complete example of a filter that fetches the id, title, and date of every story the site published on a Tuesday.

        $ curl -s https://feeds.npr.org/1019/feed.json |
jq '.items | .[] |
  select(
    .date_published |
    .[0:19] + "Z" |
    fromdate |
    strftime("%a") == "Tue"
  ) |
  {id: .id, title:.title, date:.date_published}'

After selecting the items property, this filter uses .[] to iterate over each item. The main bulk of the filter uses the select function to keep only those posts with a date_published value whose weekday (strftime("%a")) is Tue. The strftime function requires a very specifically formatted date which the filter .[0:19] + "Z" constructs.

After selecting the desired items, the final filter builds an object for each one with the required fields. Note that every time the feed changes, the results will differ. Here’s an example taken at the time of publication:

        {
  "id": "959667930",
  "title": "Deplatforming: Not A First Amendment Issue, But Still A Tough Call For Big Tech",
  "date": "2021-01-26T17:00:00-05:00"
}
{
  "id": "960679189",
  "title": "My Pillow CEO Mike Lindell Permanently Suspended From Twitter",
  "date": "2021-01-26T10:17:15-05:00"
}
{
  "id": "960220477",
  "title": "Is Your 401(k) Colluding To Make Cereal More Expensive?",
  "date": "2021-01-26T06:31:24-05:00"
}

How to Process JSON Online Using jqplay

If you want to try jq out before downloading it, jqplay is the perfect place to start. Via a simple interface, the site allows you to enter sample JSON and a filter, then view the result.

It also allows you to try out a few different options. These include --compact-output (to remove whitespace) and --null-input (to show the result on missing input).

The interface also includes a very useful cheatsheet section. Here’s a screenshot of the previous long example:

A screenshot of the jq play online playground

Note that, as with that link, you can also share examples via a URL.

Use jq to Read and Manipulate JSON Data

You can find full information about jq from the tutorial and manual, both of which live on the jq website. The program itself offers a limited amount of help via the --help option.

If you want to carry out basic filters and transformations or read a large chunk of JSON, jq is a valuable tool.