Advancing Spark – Runtime 8 2 and Advanced Schema Evolution

Another week, another new Databricks Runtime.

Runtime 8.2 brings some nice functionality around operational metrics, but the big star of the week is the new Schema Inference & Evolution functionality available through Autoloader.

In this video, Simon takes a look through simple schema inference, applying schema hints and watching the schema metadata evolve through the various config options available!

Frank

#DataScientist, #DataEngineer, Blogger, Vlogger, Podcaster at http://DataDriven.tv . Back @Microsoft to help customers leverage #AI Opinions mine. #武當派 fan. I blog to help you become a better data scientist/ML engineer Opinions are mine. All mine.