Basic JSON Wrangling with JQ

Introduction

In the world of data, JSON has become the lingua franca for structuring and exchanging information. Consequently, it’s common to encounter and work with JSON in some way, and this nifty little program, jq, is the perfect helper for such tasks. The jq program is a lightweight and flexible command-line JSON processor that can boost your productivity. Since it’s a command-line processor, it means that we can, for example, pipe JSON from a file or from a curl request:

curl 'https://example.com/cats' | jq '<command>'

By using the pipe (|) operator, we can effectively integrate jq with other command-line tools in order to traverse nested JSON structures in combination with filtering, sorting and transformations. All with the goal of making whatever JSON data more understandable.

Installation

Depending on your operating system, install jq through one of the package managers:

Mac (using Homebrew)

brew install jq

Linux Ubuntu

sudo apt-get install jq

If you are on another operating system or have another package manager of choice, you can go read the installation guides for alternatives. Or if you rather not install it right now, there is also a jq playground.

Usage

By default, jq will pretty-print all output neatly for us. So the most simple filter available, the identity operator (.), validates the input and produces the same value as output pretty-printed. Here is an example that formats a curl JSON output:

curl 'https://example.com/api/cats' | jq '.'

However it’s useful to know that the . is not needed for the identity filter. Only jq will suffice.

Filtering

By making use of some of jq’s built-in functions we can quite easily filter arrays. First, here’s a basic example:

echo '[1,2,3,4]' | jq 'map(select(. % 2 == 0))'
# Output: [2, 4]

If we add more information to the input data by introducing an array that holds objects, we may filter on a selected attribute:

echo '[{"name": "paul", "isAlive": true}, {"name": "john", "isAlive": false}, {"name": "ringo", "isAlive": true}, {"name": "george", "isAlive": false}]' | jq 'map(select(.alive))'

Executing the last command would render an pretty-printed array containing the objects of both Paul and Ringo.

Sorting

There are also functions for sorting data - sort and sort_by. For another basic example where the input data is an array of numbers, sort is the preferred choice:

echo '[4,2,1,3]' | jq 'sort'
# Output: [1,2,3,4]

But as with filtering, it is also possible to sort by an attribute using sort_by:

echo '[{"name": "paul", "songs": 180}, {"name": "john", "songs": 180}, {"name": "ringo", "songs": 11}, {"name": "george", "songs": 25}]' | jq 'sort_by(.songs)'

Both outputs produced by the above examples will output the result in ascending order. The order can be flipped by piping on the reverse function as in jq 'sort_by(.songs) | reverse'.

Conclusion

This post only scratches the surface of what jq is and how it can be useful, but there are many more functions and operators beyond filtering and sorting that can assist when facing more complex input data. Hopefully, this post sparked some interest because I think jq can be a real productivity booster. Happy wrangling!