Advancing Analytics takes a closer look at the two new runtimes available for Databricks.
We have not just one but two new Databricks Runtimes currently in preview – 7.6 brings several new features focussing on making Autoloader more flexible, improving performance of Optimize and Structured Streaming.
Runtime 8.0 is a much wider change, seeing the shift to Spark 3.1, introducing new language versions for Python, Scala and R.
This shift brings a large swathe of functionality, performance and feature changes, so take some time to look through the docs.
Simon walks through the high level notes, pulling out some interesting features and improvements.