Here’s an interesting documentary (“Canada – The Rise of AI”, Ep. 11) on the “Canadian Silicon Valley.”

Silicon Valley may be home to some of the biggest tech giants in the world but it’s being challenged like never before. Crazy tech geniuses have popped up all over the planet making things that will blow your mind. Author and journalist Ashlee Vance is on a quest to find the most innovative tech creations and meet the beautiful freaks behind them. Bloomberg Businessweek presents an exclusive premiere of the latest episode of Hello World, the tech-travel show hosted by journalist and best-selling author Ashlee Vance and watched by millions of people around the globe.

In this video Chris Seferlis gives an overview of Azure Synapse Link, a newer feature of the Synapse Analytics Suite of tools.

Find out why this feature is important, the way it moves Operational Data to Analytical Data, and what you can then do with it.

More details about the service and some great tutorials can be found here:

Big Data Engineering closely examines  Spark Standalone Architecture.

Apache Spark has a well-defined layered architecture where all the spark components and layers are loosely coupled. This architecture is further integrated with various extensions and libraries. Apache Spark Architecture is based on two main abstractions:
Resilient Distributed Dataset (RDD)
Directed Acyclic Graph (DAG)

Chris Seferlis introduce us to the newly added Apache Spark Pools in Azure Synapse Analytics for Big Data, Machine Learning, and Data Processing needs.

From the description:

I give an overview of what Spark is, and where it came from; why the Synapse Team added it to the suite of offering, and some sample workloads why you might use it.In this video I introduce the newly added Apache Spark Pools in Azure Synapse Analytics for Big Data, Machine Learning, and Data Processing needs. I give an overview of what Spark is, and where it came from; why the Synapse Team added it to the suite of offering, and some sample workloads why you might use it.

Delta Lake is an open-source storage management system (storage layer) that brings ACID transactions and time travel to Apache Spark and big data workloads.

The latest and greatest of Delta Lake 0.7.0 requires Apache Spark 3 and among the features is a full coverage of SQL DDL and DML commands.