Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I love idea of "json lines" which is implicitly here, in jq. You can convert giant json array of objects to "json lines" and back with jq. A lot of tools could produce them, and then you can mix and match jq, simple unix grep, head etc, because each unix line of your stream is a complete parsable json, guaranteed.

back in a day I've loaded a lot of data to postgresql and elastic search after preprocessing it with very simple but powerful chain of CSV parsers (I've used CSVfix), jq, sort, grep, etc.



This concept has a name: "ndjson", for "newline-delimited JSON". http://ndjson.org/

Some tools from the Node.js universe. `npm install $etc`. https://www.npmjs.com/package/ndjson-cli


I use https://www.npmjs.com/package/concatjson to get a Stream interface in node.


json-lines is as much of a name for the concept as ndjson is

https://jsonlines.org https://github.com/ndjson/ndjson.github.io/issues/1


ndjson has more than a few times saved the scalability concerns of my technical designs.


Thanks, codesnik -- good tip!

My favorite tool in this area is `lnav` (https://lnav.org), a scriptable / chainable mini-ETL with embedded sqlite.

Oh and tiny correction (intended as helpful not nit-picking), the idiom is "back in the day" not "back in a day". :)


It sure would be cool if all the common Unix tools had JSON lines / ndjson output mode.

It could also be done as a library of adapters. e.x. something like

    j ls
Would run “ls”, parse the output, and re-output it as JSON lines.


..not sure if a plug but this does exactly that: https://github.com/kellyjonbrazil/jc


That's awesome thank you!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: