jq and yq: transforming embedded key-value strings in YAML

A YAML file with key-value pairs jammed into a single string field, info: height=170 age=25 gender="female" school="MIT University", needs a proper structure before anything downstream can use it. yq moves between YAML and JSON; jq does the actual reshaping. Together they turn that flat string into typed fields.

The input and the target

# input.yml
name: Kate
info: height=170 age=25 gender="female" school="MIT University"

# desired output
name: Kate
height: "170"
age: "25"
gender: female
school: MIT University

Building the pipeline

Convert to JSON first, since jq doesn’t speak YAML:

yq -o json input.yml > input.json

The transformation itself is one scan over the info string, pulling out key=value and key="quoted value" pairs, followed by from_entries to turn the array of pairs into an object:

.info |
[scan("(?<key>[^ =]+)=(?<value>\"[^\"]+\"|[^\" ]+)") |
{
  key: .[0],
  value: .[1]|(if startswith("\"") then .[1:-1] else . end)
}] |
from_entries

• (?<key>[^ =]+)= captures everything before = as the key
• (?<value>\"[^\"]+\"|[^\" ]+) captures either a quoted value or a bare token
• the if startswith strips the surrounding quotes when present
• from_entries converts [{key:, value:}, ...] into a plain object

Merged into a single pipeline, replacing info with its parsed fields:

yq -ojson input.yml | \
  jq '.+(.info|[scan("(?<key>[^ =]+)=(?<value>\"[^\"]+\"|[^\" ]+)") | \
  {key: .[0], value: .[1]|(if startswith("\"") then .[1:-1] else . end)}]|from_entries)|del(.info)' | \
  yq -P

Handling more than strings

The same scan doesn’t know that "98.5" should be a number or "true" should be a boolean. Add the type coercion into the value branch:

value: .[1]|(
  if startswith("\"") then .[1:-1]
  elif . == "true" then true
  elif . == "false" then false
  elif test("^[0-9]+$") then tonumber
  elif test("^[0-9]+\\.[0-9]+$") then tonumber
  else .
  end
)

And it works the same way one level down, inside an array, by mapping the same expression over .members:

yq -ojson input.yml | \
  jq '.members |= map(.+(.info|[scan("(?<key>[^ =]+)=(?<value>\"[^\"]+\"|[^\" ]+)") | \
  {key: .[0], value: .[1]|(if startswith("\"") then .[1:-1] else . end)}]|from_entries)|del(.info))' | \
  yq -P

Notes from using this on real data

For anything beyond a one-off, put the jq filter in its own file rather than inlining it in a shell pipeline. It’s easier to read, and you can give it a name:

# transform.jq
def process_info:
  [scan("(?<key>[^ =]+)=(?<value>\"[^\"]+\"|[^\" ]+)") |
  {key: .[0], value: .[1]|(if startswith("\"") then .[1:-1] else . end)}] |
  from_entries;

.+(.info|process_info)|del(.info)

Validate the input before trusting the pipeline on it (yq eval 'type' input.yml), and if a large file makes debugging the pipeline painful, break it apart with tee so you can see the JSON at each stage rather than guessing where it went wrong.