opensource

the open source projects i work on and advocate for.

the data community

apache spark · data processing framework · 44,000 stars

the most widely deployed data processing framework there is. i found a bug where spark silently dropped rows, fixed in spark 4.1.0 and 4.0.1, and i keep the most maintained fork of its go client.

apache arrow · columnar memory format · the go library

the memory format under spark, pandas, duckdb and most of modern data processing. i found a 5,000-times slowdown in its go json reader, pinned the cause with its maintainers, and did the qa on their fix, merged two days later. the same hunt found the rows spark was dropping.

databricks · data and ai platform

the data and ai platform of the fortune 500. i made data access 1,575 times faster than its own go sdk, fixed the terraform provider’s default aws policy so new workspaces deploy, and wrote unofficial clients for it in go and rust.

parquet-go · parquet for go

the parquet library for go. i identified a panic and a silent corruption in its reader, and worked both through to a fix with its maintainer.

lake-orm · lakehouse orm · mine

an orm for building on the lakehouse in go, over spark connect, for databricks, delta and iceberg. i wrote it, and the projects around it.

the programming community

gocraft/work · background jobs · 2,500 stars

durable background jobs for go on redis, sidekiq’s counterpart. i fixed its build and module path so the library could be imported again, and i ran it in production at 60,000 jobs a second.

elixir (language) · programming language · 27,000 stars

the elixir programming language, on the erlang vm. i wrote the missing use case into one of its most used functions, and its creator merged it.

parfit · comment reflow · mine

a comment reflow tool for codebases: the prose in comments rewrapped with optimal-fit line breaking, the directives a machine reads left alone. i wrote it in rust, after adam m. costello’s par.

nordlayer-nix · vpn on nixos · mine

nordsecurity’s business vpn, running on nixos. i wrote the module that gives a debian-only daemon a home it believes in.

what i advocate for

aegis · archive tool · advocate

an archive tool to help immortalize photos, documents, and other data you want to keep for your lifetime. alex david is one of my heroes.