databricks

databricks, the data and ai platform of the fortune 500.

the go sdk. i made data access 1,575 times faster than databricks’ own go sdk. reading a large result through the official sdk ran at about 55 rows a second: 100,000 rows in 30 minutes. i built a prototype in rust over arrow-odbc that pulls the result in columnar batches through a buffer it reuses, with no allocation per row: 21 million rows in 212 seconds, about 99,000 rows a second. i published the comparison on databricks’ tracker with the benchmarks and a complete, production-ready go program: buffered ndjson streaming, a sync.Pool recycling memory, zero allocations per row. a blueprint for fixing the tooling. it is the same work that reverse-engineered the java toolchains into go at reality defender, and the trail led to the root of the problem in apache arrow.

the terraform provider. i identified and fixed a problem where the provider’s default aws cross-account policy lacked the network permissions a workspace needs, so a deployment that followed the documentation failed with “failed credentials validation checks: allocate address”. i wrote the fix with positive and negative unit tests. it was merged into v1.53.0 in october 2024, and every new workspace since has carried it.

the clients. databricks-connect-go is the unofficial library that gives go programs spark connect sessions on databricks clusters: oauth tokens refreshed before they expire, a stopped cluster started on demand, sessions reused across a program. databricks-sql-warehouse-client-rs is a rust client for the sql warehouse’s statement execution api: run a statement, poll it to the end, and page through the result chunk by chunk, inline or through external links, so the caller sets the pace of ingestion.