AI Language Models & Transformers
Plausible text generation has been around for a couple of years, but how does it work – and what’s next? Rob Miles on Language Models and Transformers.
Read More
Transformers, explained: Understand the model behind GPT, BERT, and T5
-
Frank
- June 18, 2022
- AutoML
- automl transformer
- Bert
- Dale Markowitz
- GDS: Yes
- Google Cloud
- how do transformers work
- Machine Learning
- Making with Machine Learning
- Making with ML
- ML
- pr_pr: Google Cloud
- purpose: Educate
- series: Making with Machine Learning
- transformer
- transformer model
- transformer models
- Transformers
- transformers explained
- transformers machine learning
- transformers ml
- type: DevByte+
- understanding transformers
Curious how an ML model could write a poem or an op ed? Transformers can do it all. In this episode of Making with ML, Dale Markowitz explains what transformers are, how they work, and why they’re so impactful. Over the past few years, Transformers, a neural network architecture, have completely transformed state-of-the-art natural language […]
Read More
AlphaCode Explained: AI Code Generation
-
Frank
- February 15, 2022
- AI
- AI AlphaCode
- AI Codex
- Alpha Code
- AlphaCode
- AlphaCode AI
- artificial intelligence
- attention
- billion
- billion parameter
- codex
- codex ai
- Deep Mind
- DeepMind
- explained
- Explanation
- gopher explained
- Gopher Model
- GPT-2
- gpt-3
- gpt-4
- language modeling
- Large Language Model
- logic reasoning
- Machine Learning
- ML
- Natural Language Processing
- NLP
- OpenAI
- OpenAI Codex
- Retro
- self attention
- Self-Attention
- Self-Supervised Learning
- sota
- State of the Art
- Text Generation
- transformer
AlphaCode is DeepMind’s new massive language model for generating code. It is similar to OpenAI Codex, except for in the paper they provide a bit more analysis. The field of NLP within AI and ML has exploded get a lot more papers all the time. Hopefully this video can help you understand how AlphaCode works […]
Read More
Geometric Deep Learning: The Erlangen Programme of ML
-
Frank
- July 6, 2021
- AI
- artificial intelligence
- Cancer
- CNN
- Computer Graphics
- Computer Vision
- Convolutional Neural Networks
- Deep Learning
- drug design
- equivariance
- erlangen program
- geometric deep learning
- Geometry
- GNN
- graph learning
- graph neural networks
- group theory
- hyperfoods
- immunotherapy
- invariance
- Machine Learning
- manifld learning
- Neural Network
- positional encoding
- Proteins
- symmetry
- transformer
- Transformers
The ICLR 2021 Keynote “Geometric Deep Learning: The Erlangen Programme of ML“ by Michael Bronstein is presented below.
Read More
GLOM: How to represent part-whole hierarchies in a neural network (Geoff Hinton’s Paper Explained)
-
Frank
- March 3, 2021
- AI
- artificial intelligence
- Arxiv
- attention mechanism
- Capsule Networks
- capsule networks explained
- column
- Computer Vision
- consensus algorithm
- Deep Learning
- deep learning tutorial
- explained
- Geoff Hinton
- geoff hinton capsule networks
- geoff hinton neural networks
- Geoffrey Hinton
- geoffrey hinton deep learning
- geoffrey hinton glom
- glom model
- Google AI
- Google Brain
- hinton glom
- introduction to deep learning
- Machine Learning
- Neural Networks
- Schmidhuber
- transformer
Yannic Kilcher covers a paper where Geoffrey Hinton describes GLOM, a Computer Vision model that combines transformers, neural fields, contrastive learning, capsule networks, denoising autoencoders and RNNs. GLOM decomposes an image into a parse tree of objects and their parts. However, unlike previous systems, the parse tree is constructed dynamically and differently for each input, […]
Read More){kind=link}
Transformers for Image Recognition at Scale
-
Frank
- October 6, 2020
- AI
- andrej karpathy
- anonymous
- artificial intelligence
- Arxiv
- attention is all you need
- attention mechanism
- beyer
- big transfer
- bit
- CNN
- Convolutional Neural Network
- Data Science
- Deep Learning
- explained
- Google Brain
- google research
- iclr
- iclr 2021
- karpathy
- Machine Learning
- Neural Networks
- Paper
- peer review
- review
- TPU
- tpu v3
- transformer
- transformer computer vision
- transformer images
- under submission
- vaswani
- vision transformer
- visual transformer
- vit
Yannic Kilcher explains why transformers are ruining convolutions. This paper, under review at ICLR, shows that given enough data, a standard Transformer can outperform Convolutional Neural Networks in image recognition tasks, which are classically tasks where CNNs excel. In this Video, I explain the architecture of the Vision Transformer (ViT), the reason why it works […]
Read More
Explaining the Paper: Hopfield Networks is All You Need
-
Frank
- August 10, 2020
- AI
- artificial intelligence
- Arxiv
- attention
- attention is all you need
- Bert
- binary
- continuous
- Deep Learning
- energy function
- error
- explained
- exponental
- gru
- hochreiter
- hopfield
- hopfield network
- key
- lse
- LSTM
- Machine Learning
- metastable
- Neural Networks
- Paper
- pattern
- query
- retrieval
- RNN
- routing
- Schmidhuber
- separation
- store
- transformer
- update rule
- value
Yannic Kilcher explains the paper “Hopfield Networks is All You Need.” Hopfield Networks are one of the classic models of biological memory networks. This paper generalizes modern Hopfield Networks to continuous states and shows that the corresponding update rule is equal to the attention mechanism used in modern Transformers. It further analyzes a pre-trained BERT […]
Read More
14 Cool Apps Built on OpenAI’s GPT-3 API
Bakz T. Future shows off 14 Cool applications built on top of OpenAI’s GPT-3 (general purpose transformer) API (currently in private beta).
Read More
Object-Centric Learning with Slot Attention
-
Frank
- July 1, 2020
- AI
- artificial intelligence
- Arxiv
- attention
- attention mechanism
- capsules
- clevr
- CNN
- Convolutional Neural Network
- Deep Learning
- detr
- disentanglement
- embeddings
- encoder
- ethz
- explained
- gru
- LSTM
- Machine Learning
- Neural Networks
- objects
- Paper
- permutation invariant
- render
- routing
- set
- slots
- tetris
- transformer
- vision
- weight sharing
Visual scenes are often comprised of sets of independent objects. Yet, current vision models make no assumptions about the nature of the pictures they look at. Yannic Kilcher explore a paper on object-centric learning. By imposing an objectness prior, this paper a module that is able to recognize permutation-invariant sets of objects from pixels in […]
Read More
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
-
Frank
- October 7, 2019
- attention
- attention layers
- Bert
- bidirectional
- Deep Learning
- elmo
- embeddings
- fine tuning
- Jacob Devlin
- Kenton Lee
- Kristina Toutanova
- language
- language modeling
- Machine Learning
- Ming-Wei Chang
- Natural Language Processing
- NLP
- pretrained
- squad
- transformer
- unsupervised
- word vectors
- wordpiece
- Yannic Kilcher
Yannic Kilcher investigates BERT and the white paper associated with it https://arxiv.org/abs/1810.04805 Abstract:We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers. As […]
Read More