Mastering jq — The Complete Hands-On JSON Processor Guide

Foreword

Modern development, automation, and DevOps workflows rely heavily on JSON. Configuration files, REST API responses, and structured logs are all JSON.
Enter jq — a lightweight yet powerful command-line tool that lets you slice, filter, map, and reshape JSON with surgical precision.
This guide takes you step-by-step from basic filters to advanced functional constructs so that you can confidently parse and transform data right in your terminal.

Every concept is paired with real examples and expected output. You’ll learn the logic behind each filter, not just the syntax. By the end, jq will feel as natural as grep or awk.

1. Introduction to jq

jq is short for “JSON Query.” It reads JSON input, applies a filter expression, and writes transformed output.
Its design philosophy is functional: data flows through filters and pipes, just like Unix commands.

echo '{"name":"Alice","age":30}' | jq '.name'

Output:
"Alice"

Unlike traditional text-processing tools such as grep or sed, jq understands JSON structure.
You don’t use regex to find fields — you navigate them.

Key Advantages of jq

✅ Native JSON awareness — no parsing errors due to punctuation or whitespace.
✅ Declarative filters — describe what you want, not how to loop.
✅ Chainable operations — combine transformations like pipes in Bash.
✅ Cross-platform — works on Linux, macOS, and Windows.

Basic jq Workflow

Feed JSON input via a file or standard input (stdin).
Specify a filter expression (e.g. .users[].name).
View the formatted JSON output.

cat users.json | jq '.users[].name'

Output:
"Alice"
"Bob"
"Charlie"

Understanding Filters

In jq, the dot (.) represents the current JSON value.
Every filter takes an input and produces an output. Filters can be chained together using the pipe operator (|).

jq '. | keys'

Output:
["name","age"]

This returns all keys at the top level of the JSON object.
You can think of . as a variable holding the current object being processed.

🚀 Advanced Note: Filters are composable functions.
For example, .users | map(.name) means “from the users array, extract the name of each element.”

jq vs Traditional Tools

Goal	Traditional Command	jq Equivalent
Find a name field	grep -o ‘”name”: “[^”]*”‘ file.json	jq ‘.name’
Pretty-print JSON	cat file.json \| python -m json.tool	jq .
Sum numbers in array	awk ‘{sum+=$1} END{print sum}’	jq ‘add’

Quick Example: Pretty Printing

curl -s https://api.github.com/repos/stedolan/jq | jq .

jq automatically indents and colorizes output when run interactively.
This makes it invaluable for debugging APIs and configs.

⚠️ Note: Colors depend on your terminal. Use jq -M to disable color output in scripts.

Summary

jq processes JSON structurally, not textually.
Filters define transformations.
Pipes chain filters to build complex operations.

2. Installing jq

jq is lightweight, open-source, and pre-packaged for most Unix-like systems. Installation methods vary depending on your platform.

Linux

sudo apt update
sudo apt install jq

macOS

brew install jq

Windows

choco install jq

💡 Tip: To verify installation, run jq --version. The output should resemble jq-1.6.

jq --version
Output:
jq-1.6

3. Basic Syntax and Filters

The jq filter defines how the input JSON is transformed.
At its simplest, a filter is just a reference to a field, such as .name or .user.id.

The Identity Filter

The dot (.) represents the entire JSON input. It is the identity filter — it returns its input unchanged.

echo '{"x":1,"y":2}' | jq .

Output:
{
  "x": 1,
  "y": 2
}

Accessing Object Properties

You can navigate JSON objects using dot notation, similar to JavaScript.

echo '{"user":{"name":"Alice","email":"[email protected]"}}' | jq '.user.name'

Output:
"Alice"

Nested fields are accessible by chaining dots:

.user.name

Accessing Array Elements

Arrays are indexed starting at zero. Use brackets to access elements by index.

echo '[10, 20, 30]' | jq '.[1]'

Output:
20

Iterating Over Arrays

To process every element, use .[] — it unpacks each array element sequentially.

echo '[10, 20, 30]' | jq '.[]'

Output:
10
20
30

Each element is streamed through the filter pipeline individually.

Accessing Nested Arrays and Objects

echo '{"users":[{"name":"Alice"},{"name":"Bob"}]}' | jq '.users[].name'

Output:
"Alice"
"Bob"

Extracting Multiple Fields

You can use the object constructor syntax to extract multiple fields at once.

echo '{"name":"Alice","age":30,"email":"[email protected]"}' | jq '{n:.name, e:.email}'

Output:
{
  "n": "Alice",
  "e": "[email protected]"
}

Chaining Filters

Filters can be combined using the pipe (|) operator. The output of the left-hand side becomes the input for the right-hand side.

echo '{"numbers":[1,2,3,4]}' | jq '.numbers | map(. * 2)'

Output:
[2,4,6,8]

Filtering Specific Elements

echo '[{"name":"Alice","age":30},{"name":"Bob","age":40}]' | jq '.[] | select(.age > 35)'

Output:
{
  "name": "Bob",
  "age": 40
}

Boolean and Comparison Operators

jq supports the standard set of logical and comparison operators:

== Equal
!= Not equal
> Greater than
< Less than
and, or, not

echo '{"a":5,"b":10}' | jq '.a < .b'

Output:
true

4. Accessing Objects and Arrays — In Depth

jq treats arrays and objects as first-class data structures. You can extract, modify, and rebuild them declaratively.

Creating New JSON Objects

echo '{"first":"Alice","last":"Smith"}' | jq '{fullName: (.first + " " + .last)}'

Output:
{
  "fullName": "Alice Smith"
}

Reordering Fields

You can control the order of keys when constructing new objects:

echo '{"a":1,"b":2,"c":3}' | jq '{c:.c, a:.a}'

Output:
{
  "c": 3,
  "a": 1
}

Combining Objects

jq uses the + operator to merge objects:

echo '{"a":1}' '{"b":2}' | jq -s 'add'

Output:
{
  "a": 1,
  "b": 2
}

Flattening Arrays

echo '[[1,2],[3,4]]' | jq 'add'

Output:
[1,2,3,4]

Understanding Data Flow

Each filter receives an input (often the current JSON value) and produces an output. You can visualize filters as data pipelines.

Practical Example

Suppose you have a file employees.json:

[
  {"name":"Alice","salary":5000},
  {"name":"Bob","salary":6000},
  {"name":"Charlie","salary":5500}
]

cat employees.json | jq '.[] | {employee: .name, annual: (.salary * 12)}'

Output:
{
  "employee": "Alice",
  "annual": 60000
}
{
  "employee": "Bob",
  "annual": 72000
}
{
  "employee": "Charlie",
  "annual": 66000
}

This pattern — iterate, compute, rebuild — is foundational to jq scripting.

5. Pipes and Filter Composition

The pipe operator (|) is jq’s secret weapon. It lets you send the output of one filter as the input of another, just like Unix pipelines.
This is the foundation for composing complex transformations from small, readable pieces.

echo '{"numbers":[1,2,3,4]}' | jq '.numbers | map(. * 3) | add'

Output:
30

Let’s break that down:

.numbers extracts the array.
map(. * 3) multiplies each element by three.
add sums all elements in the array.

Combining Filters Logically

Use commas (,) to produce multiple outputs from a single input:

echo '{"a":1,"b":2}' | jq '.a, .b'

Output:
1
2

Using Parentheses to Group Operations

echo '{"numbers":[1,2,3,4]}' | jq '(.numbers | map(. * 2)) | add'

Output:
20

Parentheses ensure filters are evaluated in the desired order, especially when mixing pipes and arithmetic.

6. Selectors and Conditions

The select() function filters arrays based on a condition.
Only elements that evaluate to true for the condition are kept.

echo '[{"name":"Alice","age":25},{"name":"Bob","age":35}]' | jq '.[] | select(.age > 30)'

Output:
{
  "name": "Bob",
  "age": 35
}

Multiple Conditions

echo '[{"n":"A","a":20},{"n":"B","a":30},{"n":"C","a":40}]' | jq '.[] | select(.a > 25 and .a < 40)'

Output:
{
  "n": "B",
  "a": 30
}

Negation

echo '[1,2,3,4,5]' | jq '.[] | select(. % 2 != 0)'

Output:
1
3
5

⚠️ Common Pitfall: Don’t forget that jq’s logical operators use lowercase: and, or, not.

Conditional Expressions

jq includes if / then / else constructs for inline logic.

echo '{"score":85}' | jq 'if .score >= 90 then "A" elif .score >= 80 then "B" else "C" end'

Output:
"B"

7. map, select, and reduce

Three of jq’s most powerful functional primitives are map, select, and reduce.
Together they let you transform and aggregate data without explicit loops.

map()

map applies a filter to every element of an array and returns a new array with the results.

echo '[1,2,3,4]' | jq 'map(. * 10)'

Output:
[10,20,30,40]

Chaining map with select

echo '[1,2,3,4,5,6]' | jq 'map(select(. > 3))'

Output:
[4,5,6]

reduce

reduce iterates through an array, maintaining an accumulator.
It’s similar to fold in functional programming.

echo '[1,2,3,4,5]' | jq 'reduce .[] as $num (0; . + $num)'

Output:
15

reduce with condition

echo '[{"age":20},{"age":25},{"age":30}]' | jq 'reduce .[] as $p (0; if $p.age > 20 then .+1 else . end)'

Output:
2

Nested map and reduce

echo '[{"group":[1,2]},{"group":[3,4]}]' | jq 'map(.group | add) | add'

Output:
10

Here, each group array is summed individually, and then all sums are added together.

🚀 Advanced Note: You can combine reduce with map to create custom aggregations — e.g., computing averages or combining nested objects.

Building Aggregations

Example: find the total salary of employees older than 30.

echo '[{"name":"A","age":25,"salary":4000},{"name":"B","age":35,"salary":6000}]' | jq 'reduce .[] as $e (0; if $e.age > 30 then . + $e.salary else . end)'

Output:
6000

map_values

map_values applies a filter to each value of an object (not array):

echo '{"a":1,"b":2,"c":3}' | jq 'map_values(. * 2)'

Output:
{
  "a": 2,
  "b": 4,
  "c": 6
}

walk()

walk recursively applies a transformation to all elements of an array or object.

echo '{"x":[1,2,3],"y":{"z":4}}' | jq 'walk(if type == "number" then . * 10 else . end)'

Output:
{
  "x": [10,20,30],
  "y": {"z":40}
}

💡 Tip: walk is part of jq 1.6. Use it to avoid writing repetitive recursive filters.

8. Transforming JSON Structures

One of jq’s biggest strengths is its ability to reshape JSON — to reorganize, filter, or enrich complex data into a new form.
You can create new objects, flatten nested structures, or extract specific keys into new arrays.

Renaming Keys

echo '{"first":"Alice","last":"Smith"}' | jq '{fullName: (.first + " " + .last)}'

Output:
{
  "fullName": "Alice Smith"
}

Changing Data Shape

You can move data between arrays and objects easily using filters and constructors.

echo '{"users":[{"name":"Alice","id":1},{"name":"Bob","id":2}]}' | jq '.users | map({(.id|tostring): .name}) | add'

Output:
{
  "1": "Alice",
  "2": "Bob"
}

💡 Tip: The add filter merges multiple small objects into one large object.

Grouping Data

jq 1.6 introduces the group_by function — useful for reorganizing arrays based on a key.

echo '[{"dept":"IT","name":"Alice"},{"dept":"HR","name":"Bob"},{"dept":"IT","name":"Charlie"}]' |
jq 'group_by(.dept) | map({(.[0].dept): map(.name)}) | add'

Output:
{
  "HR": ["Bob"],
  "IT": ["Alice","Charlie"]
}

Flattening Nested Objects

echo '{"a":{"b":{"c":42}}}' | jq 'paths(scalars) as $p | {"path":$p,"value":getpath($p)}'

Output:
{
  "path": ["a","b","c"],
  "value": 42
}

🚀 Advanced Tip: Use getpath and setpath for precise control of nested structures when automating configuration transforms.

9. Variables and Parameters

jq allows you to pass external values into filters using the --arg and --argjson flags.
This is invaluable when integrating jq inside shell scripts or when building parameterized data pipelines.

Using –arg

jq --arg name "Alice" '.user = $name' file.json

Input (file.json):
{}
Output:
{
  "user": "Alice"
}

--arg always passes strings. If you need numeric or structured data, use --argjson.

Using –argjson

jq --argjson limits '{"cpu":4,"mem":8}' '.config = $limits' base.json

⚙️ Note: Values set with $variables are immutable within a single jq invocation; they act like constants.

Inline Variables with as

You can create internal variables using as.

echo '{"a":10,"b":20}' | jq '. as $in | {"sum": ($in.a + $in.b)}'

Output:
{
  "sum": 30
}

Looping with foreach

foreach is like reduce, but you can output intermediate results as you iterate.

echo '[1,2,3]' | jq 'foreach .[] as $x (0; . + $x; .)'

Output:
1
3
6

Dynamic Key Names

You can use interpolation to build dynamic keys.

echo '{"id":42,"name":"Server"}' | jq '{("host_" + (.id|tostring)): .name}'

Output:
{
  "host_42": "Server"
}

10. Built-in and Custom Functions

jq ships with dozens of built-in functions — string manipulation, arithmetic, array and object utilities, and even date/time helpers.
You can also define your own using the def keyword.

String Functions

length — Number of elements in an array or characters in a string.
split(",") / join(",") — Convert between strings and arrays.
startswith(), endswith(), contains()
ascii_downcase and ascii_upcase

echo '"Hello World"' | jq '. | ascii_downcase | split(" ")'

Output:
["hello","world"]

Number Functions

add, min, max, floor, ceil, sqrt, and more.

echo '[1,5,9]' | jq 'add, max'

Output:
15
9

Object and Array Utilities

has("key") — Check if an object has a key.
keys — List all keys.
unique, sort, reverse

echo '[5,3,5,1]' | jq 'unique | sort'

Output:
[1,3,5]

Defining Custom Functions

def average: add / length;
echo '[10,20,30]' | jq 'average'

Output:
20

Chaining Custom Functions

def double: map(. * 2);
def sum: add;
echo '[1,2,3]' | jq 'double | sum'

Output:
12

Function Scoping and Recursion

Functions can call themselves recursively — for example, to flatten nested arrays.

def flatten:
  if type == "array" then map(flatten) | add
  else [.] end;
echo '[1,[2,[3]]]' | jq 'flatten'

Output:
[1,2,3]

11. jq Modules

When your jq filters start to get complex, it’s better to organize them into modules — reusable files that define custom functions and logic.
Modules work just like libraries in other languages: you save them with a .jq extension and import them into your main script.

Creating a Module

# file: mathutils.jq
def square(x): x * x;
def cube(x): x * x * x;

Using a Module

jq -L . -n 'import "mathutils" as m; [m::square(4), m::cube(3)]'

Output:
[16,27]

The -L flag tells jq where to look for module files.
You can structure large jq projects this way for better maintainability.

💡 Tip: Prefix imported definitions with a namespace (m::) to avoid naming collisions.

Practical Example: Configuration Template

Imagine a configuration template generator for multiple environments:

# configlib.jq
def base: {version:"1.0",service:"api"};
def env(name): base + {env:name};

jq -L . -n 'import "configlib" as cfg; [cfg::env("dev"), cfg::env("prod")]'

Output:
[
  {"version":"1.0","service":"api","env":"dev"},
  {"version":"1.0","service":"api","env":"prod"}
]

12. Slurping, Streaming, and Large Files

By default jq processes JSON documents one at a time.
You can change this behavior using slurp mode (-s) and stream mode (--stream) for huge data sets.

Slurp Mode (-s)

In slurp mode jq reads all inputs into one array.

echo '{"a":1}' '{"b":2}' | jq -s 'add'

Output:
{"a":1,"b":2}

⚙️ When to Use: Use -s when you have many small JSON objects that should be treated as a single collection.

Stream Mode (–stream)

Stream mode parses JSON as a sequence of path/value pairs, useful for massive files that don’t fit in memory.

jq --stream 'select(length==2)' bigfile.json

Each item in the stream is an array: the first element is the JSON path, the second is the value.

Output example:
[["users",0,"name"],"Alice"]
[["users",1,"name"],"Bob"]

Streaming Example: Counting Records

jq --stream 'reduce inputs as $i (0; if ($i[0][-1] == "id") then .+1 else . end)' data.json

Output:
42

13. Input, Output, and Formatting

Pretty-Printing JSON

cat config.json | jq .

jq automatically indents and colorizes JSON for easier reading.
Use -M to disable colors, and -c for compact output.

jq -c . config.json

Raw Output

Use -r to print raw strings instead of quoted JSON values — perfect for scripting.

echo '{"url":"https://api.example.com"}' | jq -r '.url'

Output:
https://api.example.com

Output Redirection

Redirect jq output to files or feed it into other Unix utilities.

jq '.items[] | .name' data.json > names.txt

Sorting and Formatting Arrays

echo '[3,1,2]' | jq 'sort'

Output:
[1,2,3]

Compact Transformations

Combine transformations for clean pipelines:

cat users.json | jq -cr '.users[] | [.id,.name] | @csv'

Output:
1,"Alice"
2,"Bob"

Encoding Formats

@csv — Convert an array into a CSV line.
@tsv — Tab-separated.
@json — JSON string encoding.
@html — Escape HTML entities.

echo '[ "A & B", "C < D" ]' | jq -r '@html'

Output:
A & B C < D

Formatting Numbers

echo '12345.6789' | jq 'round, floor, ceil'

Output:
12346
12345
12346

Escaping and Quoting

To safely print strings inside shell scripts, combine -r and @sh:

jq -r '.path | @sh' config.json

14. Using jq in Bash Scripts

jq is designed to slot directly into Bash automation.
Because it reads from stdin and writes to stdout, you can treat it like any other filter.

Capturing jq Output in Variables

API_URL=$(jq -r '.api.url' config.json)
echo "Connecting to $API_URL..."

With -r you avoid quotes around strings, which makes assignment clean.

Processing API Responses

response=$(curl -s https://api.github.com/users/octocat)
name=$(echo "$response" | jq -r '.name')
repos=$(echo "$response" | jq -r '.public_repos')
echo "$name has $repos public repositories."

Looping Over Arrays

for repo in $(jq -r '.repos[].name' repos.json); do
  echo "Cloning $repo..."
  git clone "https://github.com/myorg/$repo.git"
done

Combining jq With Other Tools

jq pairs perfectly with grep, awk, and xargs:

jq -r '.servers[] | select(.status=="active") | .ip' inventory.json | xargs -n1 ping -c1

Embedding Multi-line Filters

For readability inside scripts, use single quotes and backslashes for line continuation.

jq '
  .users
  | map(select(.active == true))
  | map({name, email})
' users.json

💡 Tip: Always quote jq filters in scripts to prevent shell interpretation of special characters like > and $.

15. Debugging and Common Pitfalls

1. Check JSON Validity

Before debugging jq, verify your input JSON:

jq . file.json > /dev/null

If jq exits with an error, your file is not valid JSON.

2. Use the Identity Filter

If a complex filter yields nothing, try simplifying to . to confirm jq is receiving data.

3. Understand null Values

jq silently drops null results in pipelines.
If you need to keep them, wrap your expressions in arrays: [.].

4. Remember Boolean Operators

jq uses lowercase and, or, not — not shell operators like && or ||.

5. Pretty-Print Intermediate Steps

jq '.users | map(.name) | .[0]' data.json

Add | debug to trace values:

jq '.users | map(.name) | debug | .[0]' data.json

6. Don’t Forget -r for Raw Output

Quoted strings in Bash can cause issues:

FILE=$(jq '.file' conf.json)  # yields '"path/to/file"'

Fix it:

FILE=$(jq -r '.file' conf.json)

7. Handle Large Files Efficiently

Use --stream for incremental parsing.
Pipe through gzip -dc to decompress on the fly.
Filter early to minimize data.

8. Avoid Unnecessary Shell Loops

Let jq do the heavy lifting instead of spawning subshells.

# inefficient
for user in $(jq -r '.users[].name' data.json); do
  echo "$user"
done

# efficient
jq -r '.users[].name' data.json

9. Commenting jq Scripts

jq supports # comments at the start of a line.
Use them freely in long filters to maintain clarity.

10. Testing Filters Interactively

Use jqPlay (jqplay.org) to experiment and debug filters live in your browser.

🚀 Pro Tip: In complex pipelines, insert temporary debug or length filters to verify the shape of intermediate results.

Glossary

Filter: A jq expression that transforms JSON input into output.
Pipe: Operator (|) chaining filters sequentially.
map(): Applies a filter to each element of an array, returning a new array.
reduce: Iterates over an array to produce a single accumulated value.
select(): Filters elements by boolean condition.
walk(): Recursively applies a transformation to nested values.
slurp (-s): Reads all JSON inputs into a single array.
stream (–stream): Processes JSON as incremental path/value pairs.
@csv / @tsv: Format filters converting arrays into delimited text.
–arg / –argjson: Flags for injecting external values into filters.

Conclusion

By now, you’ve moved from simple field extraction to advanced functional pipelines, recursion, and streaming JSON analytics.
jq rewards experimentation — each filter you write teaches you a new way to think about structured data.
Whether you’re automating APIs, cleaning logs, or building pipelines, jq turns your terminal into a powerful data-processing engine.