apache arrow

apache arrow, the columnar memory format under spark, pandas, duckdb and most of modern data processing · the go library.

i identified a problem where arrow’s go library read json 5,000 times slower than it should. working on spark’s go client i noticed RecordFromJSON was absurdly slow, so i measured it. 10,000 records, about 29 mb, took 9 minutes and 32 seconds. NewJSONReader on the same input took 137 ms. i wrote it up with the benchmarks and the code to reproduce it, and worked it through with matt topol, an apache arrow committer. he profiled it to the json library underneath, wrote a fix, and asked me to qa it, which i did on the pull request. i benchmarked that library’s streaming decoder on its own, which pinned the root cause: it ran over a hundred times slower than decoding the same bytes in one go, and on 4 mb records it never finished. then i put the fix through production-sized data: a thousand audio transcripts of 4 mb each, about 4 gb, went through in 21 seconds. i signed it off, and it was merged that afternoon, two days after my report. every go program ingesting json through arrow got the speed-up.

the same thread ran down something worse. rows were going missing between arrow and spark, and we worked through it until the cause turned out to be in apache spark itself, which fixed it in 4.1.0 and 4.0.1.