A central challenge of current stream processors is navigating the trade-off between performance and consistency. Timely Dataflow changes that. By rethinking how time should be represented in a distributed system, it achieves lower latencies and strong consistency, while allowing its users to express even complex cyclic computations.

In 2006, Ben Moseley and Peter Marks explored something that many of us are familiar with: the creeping feeling that the complexity of a software system has outgrown the complexity of the very domain it was designed to capture. By taking a fresh look at which kinds of complexity are to be considered essential and which are merely introduced by accident, they put in words many deceptively obvious insights. Better still, they hinted at Codd’s relational model and ideas from functional programming as a way out.

Unfortunately, they left implementing their approach as an exercise to the reader, calling their own ideas “purely hypothetical” and “not proven in practice”.

What were they on to? What has changed? And how deep is the tar pit today?

The functional approach to state management in the frontend was first enabled by React, pioneered by the likes of om.next, Redux, and Elm, and has ushered in a golden era in web development. It is captured by the two signatures view :: DB -> HTML and mutate :: DB -> Tx -> DB. What might happen when we start working with more than one world, i.e. when we replace the notion of mutate with that of solve :: DB -> Tx -> [DB]? Using Clojure and Rust, we explore these ideas in the context of 3DF, a stream processing system based on differential dataflows.

3DF is a stream processing system which feeds off of Datomic’s transaction log and provides clients with the ability to register arbitrary Datalog queries, for which they will then continuously receive any changes to the result set. It does this efficiently by compiling Datalog queries to differential dataflows.

Using 3DF on top of Datomic provides a powerful, reactive interface to Datomic, making it an even more attractive choice for the real-time web. It also opens up Datomic to non-JVM runtimes and processes without a peer cache, without sacrificing performance. Finally, it hints at the possibility of significantly speeding up functional UI frameworks like D3 and React, because it allows these systems to skip their own change detection.

This talk will explore the „why?“, „how?“, and „what now?“ of working with a reactive database.

Imagine turning the way that web applications interact with the database on its head: instead of polling for changes, clients register their information interests and then continuously receive updates as new data enters the system. Can we do this while maintaining the power and flexibility of Datomic? This talk introduces 3DF, a stream processing system based on differential dataflows, which aims to do just that. Feeding off of Datomic’s transaction log, 3DF provides a reactive interface to your favourite database, making it an even more attractive choice for the next generation of web applications.