reality defender

· organisations

tech lead · product and data platform · new york · february 2024 to now.

the organisation

reality defender detects manipulated media: deepfakes in audio, video, image and text. its customers are tier-one banks, government agencies and media companies. reality defender is a y combinator company in the big apple of the xoogler ilk.

  • sxsw’s 2023 pitch award in ai and robotics, march 2023 (pr newswire).
  • a $15 million series a led by dcvc, october 2023 (pr newswire).
  • the most innovative startup of 2024 at rsa conference’s innovation sandbox, may 2024 (rsa conference).
  • the series a expanded to $33 million with ibm ventures, booz allen ventures and accenture, october 2024 (pr newswire).
  • jp morgan chase’s 2025 hall of innovation, november 2025 (pr newswire).
  • gartner’s deepfake detection company to beat, december 2025 (reality defender).

impact

  • we made data access 1,575 times faster than databricks. we then shared our findings for the world to see. the root of the problem was in apache arrow. we found it, and the maintainer had it fixed two days later (the issue).
  • we built the data platform the detection models train and are judged on, from nothing. it ingests 100 mb/s and over 100 million rows/s.
  • we serve a 100 tb warehouse of metadata to the ml team as rest endpoints. they hit a url and the data comes back. no queue, no job to poll.
  • we cut the redundant layers out of the medallion architecture (bronze does not need to exist) and drove the pipeline with my own spark connect client (the fork i maintain).
  • for the customers: soc2, hipaa and gdpr auditing, and lineage, licence and source-poisoning tracing. a model’s integrity can be shown, not asserted.

1,575 times faster

the official go sdk read a large result at about 55 rows/s. 100,000 rows took 30 minutes. the pipeline stalled and the warehouse billed for the wait.

we wrote a prototype in rust over arrow-odbc. it read 21 million rows in 212 seconds, about 99,000 rows/s, with the same queries, network and machine.

in january 2025 we filed it with databricks as a feature request: the benchmarks and a complete go program for anyone passing by. buffered ndjson streaming, a sync.Pool recycling memory, zero allocations per row.

the thread is the interesting part.

  • the first update was ours. arrow deserialisation was worth 30 to 40%. the orders of magnitude were the odbc driver’s.
  • the next was the fix in go, once databricks’ sql connector could fetch through cloud storage and arrow. we posted it so the next person would not need the rust.
  • the last came a year on, when databricks triaged it. by then the rabbit hole had led to the apache arrow pmc and a few more bugs fixed there.

the pipeline

the medallion architecture wants a bronze layer, a silver and a gold. each one is a copy and a job. bronze does not need to exist.

the spark connect client for go is my own, a fork i maintain, with the streaming the upstream one lacked. it reads the warehouse of metadata, about 100 tb, and serves it to the ml team through a strongly typed rest api with schema evolution.

the ml team never saw spark. they hit an endpoint for the slice they wanted. the backend opened a spark connect session per request, streamed the rows back as arrow, and returned when the data had.

it is synchronous. no queue, no job to poll, no table in the middle. a question and an answer.

the point of synchronous is that computation happens on demand, for the query that asks for it. there is no series of dag tasks running in a pipeline behind the scenes, the way apache airflow and its kind work. we either cache what matters or do the work up front. it is a lot simpler this way, because there is less moving machinery between you and your answer.

the same goes for writes. nobody gets a pagerduty call at night because a pipeline job failed on some bad condition. everything is strongly typed, schema and contract driven, nothing is schemaless, so breaking the pipeline is impossible: every condition is already accounted for and solved by the type system. how else do you fix pipelines that break on sloppy data, other than by not writing sloppy data?

what i did

  • the data lake and its ingestion, so the models had something to train on.
  • the rest api, so the ml team never had to learn spark.
  • the go drivers, so a read took seconds instead of half an hour.
  • ci/cd and a three-tier secured network, so soc2, hipaa and gdpr could be passed rather than promised.
  • the mlops division taught to run production-grade infrastructure. the c-suite advised on what to build and what not to.

golang, databricks, spark connect, apache arrow, terraform, aws, delta lake, unity catalog, kubernetes.

what it meant

i lead by example. i outpace everyone, whether or not they are using bots. my work merges because it is correct, so there is no bottleneck at me. i use my acumen to make sure the quality of our technology is 100% right, with zero tolerance for exceptions. i raise the bar for everyone, and i am always excited to teach people to be as good as i am at my job.

the discipline is the one i learned at geneva. everything lean. the least cumulative time spent by everyone. nothing kept that does not earn its place.