apache spark

apache spark, the most widely deployed data processing framework there is · 44,000 stars.

i found a bug where spark silently dropped rows. spark connect is how a program talks to a spark cluster without being on it, and i was building a lakehouse on it in go. a dataframe built from arrow data with more than one batch kept only the first. the rest vanished on the way into delta and iceberg tables, and nothing said so. i worked it through with matt topol, an apache arrow committer, while we chased a second bug in apache arrow itself. martin grund, the databricks principal engineer who leads spark connect, found the cause in spark’s own arrow converters. he filed it from my report as spark-53342 and fixed it. it shipped in spark 4.1.0 and was backported to 4.0.1.

i keep the most maintained fork of spark’s go client. it carries what production needs and upstream does not have yet: a database/sql driver, so goose, sqlc and anything else that speaks database/sql can run against spark; typed dataframes that collect, stream or take the first row straight into go structs; grpc dial options on the session builder; and a typed error for the cold start of a databricks serverless cluster. it is a drop-in, the session api is upstream’s, and lake-orm runs on it.