workshop on meta-learning (metalearn 2020) - sung whanyoon meta...

Support set images

CNN (𝑓𝜃)

𝚽 = 𝝓0,𝝓1,𝝓2,𝝓3

ത𝐠0 ത𝐠1 ത𝐠2ത𝐠3

𝝓𝟎′ 𝝓𝟏

′ 𝝓𝟐′

𝝓𝟑′

𝒗𝑘 = 𝝓𝑘′ − ഥ𝒈𝑘: error vector

𝐌 ← null 𝒗0, 𝒗1, 𝒗2, 𝒗3

𝝓𝑘′ = 𝑁𝑐 − 1 𝝓𝑘 − σ𝑗≠𝑘 𝝓𝑗

Meta-Learner with Linear NullingSung Whan Yoon

Postdoctoral ResearcherJun Seo

Ph. D. StudentJaekyun Moon

Professor

� An embedding network is combined with a linear transformer.� The linear transformer carries out null-space projection on an alternative classification space.� The projection space M is constructed to match the network output with a special set of reference vectors.

Images from Nc novel classes

𝐌 SoftmaxCNN(𝑓𝜃) 𝐠 𝒅(∙,∙)

𝚽

𝐌

Distance measures

Classification

Linear transformerEmbedding network

Embedding space Alternative space M Reference vectors

★: references

Oboe: Collaborative Filtering for AutoML Initialization

Chengrun Yang, Yuji Akimoto, Dae Won Kim, Madeleine Udell

Cornell University

Goal: Select models for a new dataset within time budget.

Given: Model performance and runtime on previous datasets.

Approach:

Ilow rank dataset-by-model collaborative filtering matrix

Ipredict model runtime using polynomials

Iclassical experiment design for cold-start

Imissing entry imputation for model performance prediction

Performance:

Icold-start: high accuracy

Imodel selection: fast and perform well

1 / 1

Backpropamine: meta-learning with neuromodulated Hebbian plasticity

● Differentiable plasticity: meta-learning with Hebbian plastic connections

○ Meta-train both the baseline weight and plasticity of each connection to support efficient learning in any episode

● In nature, plasticity is under real-time control through neuromodulators

○ The brain can decide when and where to be plastic

● Backpropamine = Differentiable plasticity + neuromodulation

○ Make the rate of plasticity a real-time output of the network

○ During each episode, the network effectively learns by self-modification

● Results:

○ Solves tasks that non-modulated networks cannot

○ Improves LSTM performance on PTB language modeling task

Toward Multimodal Model-Agnostic Meta-LearningRisto Vuorio1, Shao-Hua Sun2, Hexiang Hu2 & Joseph J. Lim2

University of Southern California2University of Michigan1Model-basedMeta-learner

Gradient-basedMeta-learner

x

y

( (

K⇥Samples

Task Embedding Network

�Task

Embedding

Modulation Network

Modulation Network

ModulationNetwork

x

y

�✓2⌧2

�✓1⌧1

�⌧n

✓n

…

yModel-basedMeta-learner

Gradient-basedMeta-learner

x

y

( (

K⇥Samples

Task Embedding Network

�Task

Embedding

Modulation Network

Modulation Network

ModulationNetwork

x

y

�✓2⌧2

�✓1⌧1

�⌧n

✓n…

y

The limitation of the MAML family• One initialization can be suboptimal for

multimodal task distributions.Multi-Modal MAML1. Model-based meta-learner computes

task embeddings2. Task embeddings are used to

modulate gradient-based meta-learner3. Gradient-based meta-learner adapts

via gradient steps

1. Finds architecture for CNNs in ~0.25 days

2. Based on the idea of utility of individual nodes.

3. Closely aligns with a theory of human brain ontogenesis.

Fast Neural Architecture Construction using EnvelopeNets

Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples

Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, Hugo Larochelle

● New benchmark for few-shot classification● Two-fold approach:

1. Change the data● Large-scale● Diverse2. Change the task creation○ Introduce imbalance○ Utilize class hierarchy for ImageNet

● Preliminary results on: baselines, Prototypical Networks, Matching Networks, and MAML.● Leveraging data of multiple sources remains an open and interesting research direction!

Macro Neural Architecture Search RevisitedHanzhang Hu1, John Langford2, Rich Caruana2, Eric Horvitz2, Debadeepta Dey2

1Carnegie Mellon University, 2Microsoft Research

Cell Search: applies the found template on predefined skeleton.

Macro Search: learns all connections and layer types.

Key take-away: macro search can be competitive against cell search, even with simple random growing strategies, if the initial model is the same as cell search.

Cell Search: the predefined skeleton ensures the simplest cell search can achieve 4.6% error with 0.4M params on CIFAR 10.

MetaLearn @ NeurIPS 2018 CiML @ NeurIPS 2018

AutoDL challenge design and beta testsZhengying Liu∗, Olivier Bousquet, André Elisseeff, Sergio Escalera, Isabelle Guyon, Julio Jacques Jr., Albert Clapés, Adrien Pavao, Michèle Sebag, Danny Silver, Lisheng Sun-Hosoya, Sébastien Tréguer, Wei-Wei Tu, Yiqi Hu, Jingsong Wang, Quanming Yao

Help Automating Deep Learning

Join the AutoDL challenge!https://autodl.chalearn.org

Modular meta-learning in abstract graph networksfor combinatorial generalizationFerran Alet, Maria Bauza, A. Rodriguez, T. Lozano-Perez, L. Kaelbling code&pdf:alet-etal.com

Combinatorial generalization: generalizing by reusing neural modules

Nodes tied to entities

Graph Neural Networks

Objects

Particles

Modular meta-learning

We introduce: Abstract Graph Networks

nodes are not tied to concrete entities

Graph Element Networks

Joints

OmniPush dataset

Support set

Conv BN FiLM ReLU

4x

4x

Max Pool

Query set

Conv BN FiLM ReLU Max Pool

G

Cross-ModulationNetworks For Few-Shot LearningHugo Prol†, Vincent Dumoulin‡, and Luis Herranz†

† Computer Vision Center, Univ. Autònoma de Barcelona‡ Google Brain

Key idea: allow support and query examples to interact at each level of abstraction.

☆ Channel-wise affine transformations:

Extending the feature extraction pipeline of Matching Networks:

☆ Subnetwork G predicts the affine parameters and

Large Margin Meta-Learning for Few-Shot Classification

Large Margin Principle

Fig. 1: Large margin meta-learning. (a) Classifier trained withoutthe large margin constraint. (b) Classifier trained with the largemargin constraint. (c) Gradient of the triplet loss.

L = Lsoftmax

+ � ⇤ Llarge-margin

<latexit sha1_base64="hQ7au3k0NzkRQweDozCPaQYTPD0=">AAADHHicjVFNT9tAEH1xaaH0K5RjLxahUtWqkc2lvVRC5cKBA5UIQSJRtN5swor1h9ZrBLL8V/gnvXFDXEGceykS/Q/MLo4EhKpdy+s3b+Y97+xEmZK5CYLLhvdk5umz2bnn8y9evnr9prnwdjtPC81Fh6cq1TsRy4WSiegYaZTYybRgcaREN9pfs/nugdC5TJMtc5SJfszGiRxJzgxRg2Z3uRczs8eZKjcq/9udYFD2jDg0ZZ6OTMwOq8r/5PcUOQ/Zx0fKFNNj8TmmXSZVtTxotoJ24JY/DcIatFCvzbR5gR6GSMFRIIZAAkNYgSGnZxchAmTE9VESpwlJlxeoME/agqoEVTBi92kfU7RbswnF1jN3ak5/UfRqUvp4T5qU6jRh+zff5QvnbNm/eZfO057tiL5R7RUTa7BH7L90k8r/1dleDEb46nqQ1FPmGNsdr10Kdyv25P6drgw5ZMRZPKS8JsydcnLPvtPkrnd7t8zlf7tKy9qY17UFruwpacDhw3FOg+2Vdhi0wx8rrdXv9ajn8A5L+EDz/IJVrGMTHfL+iV+4xh/v2DvxTr2z21KvUWsWcW955zcZPbSN</latexit><latexit sha1_base64="hQ7au3k0NzkRQweDozCPaQYTPD0=">AAADHHicjVFNT9tAEH1xaaH0K5RjLxahUtWqkc2lvVRC5cKBA5UIQSJRtN5swor1h9ZrBLL8V/gnvXFDXEGceykS/Q/MLo4EhKpdy+s3b+Y97+xEmZK5CYLLhvdk5umz2bnn8y9evnr9prnwdjtPC81Fh6cq1TsRy4WSiegYaZTYybRgcaREN9pfs/nugdC5TJMtc5SJfszGiRxJzgxRg2Z3uRczs8eZKjcq/9udYFD2jDg0ZZ6OTMwOq8r/5PcUOQ/Zx0fKFNNj8TmmXSZVtTxotoJ24JY/DcIatFCvzbR5gR6GSMFRIIZAAkNYgSGnZxchAmTE9VESpwlJlxeoME/agqoEVTBi92kfU7RbswnF1jN3ak5/UfRqUvp4T5qU6jRh+zff5QvnbNm/eZfO057tiL5R7RUTa7BH7L90k8r/1dleDEb46nqQ1FPmGNsdr10Kdyv25P6drgw5ZMRZPKS8JsydcnLPvtPkrnd7t8zlf7tKy9qY17UFruwpacDhw3FOg+2Vdhi0wx8rrdXv9ajn8A5L+EDz/IJVrGMTHfL+iV+4xh/v2DvxTr2z21KvUWsWcW955zcZPbSN</latexit><latexit sha1_base64="hQ7au3k0NzkRQweDozCPaQYTPD0=">AAADHHicjVFNT9tAEH1xaaH0K5RjLxahUtWqkc2lvVRC5cKBA5UIQSJRtN5swor1h9ZrBLL8V/gnvXFDXEGceykS/Q/MLo4EhKpdy+s3b+Y97+xEmZK5CYLLhvdk5umz2bnn8y9evnr9prnwdjtPC81Fh6cq1TsRy4WSiegYaZTYybRgcaREN9pfs/nugdC5TJMtc5SJfszGiRxJzgxRg2Z3uRczs8eZKjcq/9udYFD2jDg0ZZ6OTMwOq8r/5PcUOQ/Zx0fKFNNj8TmmXSZVtTxotoJ24JY/DcIatFCvzbR5gR6GSMFRIIZAAkNYgSGnZxchAmTE9VESpwlJlxeoME/agqoEVTBi92kfU7RbswnF1jN3ak5/UfRqUvp4T5qU6jRh+zff5QvnbNm/eZfO057tiL5R7RUTa7BH7L90k8r/1dleDEb46nqQ1FPmGNsdr10Kdyv25P6drgw5ZMRZPKS8JsydcnLPvtPkrnd7t8zlf7tKy9qY17UFruwpacDhw3FOg+2Vdhi0wx8rrdXv9ajn8A5L+EDz/IJVrGMTHfL+iV+4xh/v2DvxTr2z21KvUWsWcW955zcZPbSN</latexit><latexit sha1_base64="hQ7au3k0NzkRQweDozCPaQYTPD0=">AAADHHicjVFNT9tAEH1xaaH0K5RjLxahUtWqkc2lvVRC5cKBA5UIQSJRtN5swor1h9ZrBLL8V/gnvXFDXEGceykS/Q/MLo4EhKpdy+s3b+Y97+xEmZK5CYLLhvdk5umz2bnn8y9evnr9prnwdjtPC81Fh6cq1TsRy4WSiegYaZTYybRgcaREN9pfs/nugdC5TJMtc5SJfszGiRxJzgxRg2Z3uRczs8eZKjcq/9udYFD2jDg0ZZ6OTMwOq8r/5PcUOQ/Zx0fKFNNj8TmmXSZVtTxotoJ24JY/DcIatFCvzbR5gR6GSMFRIIZAAkNYgSGnZxchAmTE9VESpwlJlxeoME/agqoEVTBi92kfU7RbswnF1jN3ak5/UfRqUvp4T5qU6jRh+zff5QvnbNm/eZfO057tiL5R7RUTa7BH7L90k8r/1dleDEb46nqQ1FPmGNsdr10Kdyv25P6drgw5ZMRZPKS8JsydcnLPvtPkrnd7t8zlf7tKy9qY17UFruwpacDhw3FOg+2Vdhi0wx8rrdXv9ajn8A5L+EDz/IJVrGMTHfL+iV+4xh/v2DvxTr2z21KvUWsWcW955zcZPbSN</latexit>

One Implementation: Triplet LossLlarge-margin =

1

Nt

NtX

i=1

⇥k f�(x

ai )� f�(x

pi ) k

22 � k f�(x

ai )� f�(x

ni ) k22 +m

⇤+.

<latexit sha1_base64="M4OHnhlbxja8lJ/D5UF5g/HcIvU=">AAAD/HicnVFNb9QwEJ00fJTw0S0cuUSskFpVu0r2AAipUgUXDggViW0rbXYjJ+tkrTpOsB3UKgr/hBs3xJU/wBXOiH8A/4Kxm64oBYFwFOfNm3kvHk9ScaZ0EHx1VtwLFy9dXr3iXb12/cZab/3mniprmdJxWvJSHiREUc4EHWumOT2oJCVFwul+cvjY5PdfUalYKV7o44pOC5ILlrGUaKTidedelNCciYa+rC3VRpwklGP8UEtWcapjXirVel2dwlNpjAqiFynhzdM2biJNj3TDiczpoMCdibbdjjJJ0iZsm2exbiNVF3HDtsN2dhJzmulJVBFJOKfcz+KoWrAN65pkzVEbsxnZHPyOrjaXsng0Gw3+y0ScNdkqIsnyhZ7GW8NIlKIuEiq9iIr5sl+Dl3fkeXGvHwwDu/zzIOxAH7q1W/a+QARzKCGFGgqgIEAj5kBA4TOBEAKokJtCg5xExGyeQgseamusolhBkD3EPcdo0rECY+OprDrFv3B8JSp9uIuaEuskYvM33+Zr62zYP3k31tOc7Ri/SedVIKthgezfdKeV/6ozvWjI4IHtgWFPlWVMd2nnUttbMSf3f+pKo0OFnMFzzEvEqVWe3rNvNcr2bu6W2Pw3W2lYE6ddbQ3fzSlxwOGv4zwP9kbDMBiGz0f9nUfdqFfhNtyBDZznfdiBJ7ALY0idN85H55Pz2X3tvnXfue9PSlecTnMLziz3ww97zgl6</latexit><latexit sha1_base64="M4OHnhlbxja8lJ/D5UF5g/HcIvU=">AAAD/HicnVFNb9QwEJ00fJTw0S0cuUSskFpVu0r2AAipUgUXDggViW0rbXYjJ+tkrTpOsB3UKgr/hBs3xJU/wBXOiH8A/4Kxm64oBYFwFOfNm3kvHk9ScaZ0EHx1VtwLFy9dXr3iXb12/cZab/3mniprmdJxWvJSHiREUc4EHWumOT2oJCVFwul+cvjY5PdfUalYKV7o44pOC5ILlrGUaKTidedelNCciYa+rC3VRpwklGP8UEtWcapjXirVel2dwlNpjAqiFynhzdM2biJNj3TDiczpoMCdibbdjjJJ0iZsm2exbiNVF3HDtsN2dhJzmulJVBFJOKfcz+KoWrAN65pkzVEbsxnZHPyOrjaXsng0Gw3+y0ScNdkqIsnyhZ7GW8NIlKIuEiq9iIr5sl+Dl3fkeXGvHwwDu/zzIOxAH7q1W/a+QARzKCGFGgqgIEAj5kBA4TOBEAKokJtCg5xExGyeQgseamusolhBkD3EPcdo0rECY+OprDrFv3B8JSp9uIuaEuskYvM33+Zr62zYP3k31tOc7Ri/SedVIKthgezfdKeV/6ozvWjI4IHtgWFPlWVMd2nnUttbMSf3f+pKo0OFnMFzzEvEqVWe3rNvNcr2bu6W2Pw3W2lYE6ddbQ3fzSlxwOGv4zwP9kbDMBiGz0f9nUfdqFfhNtyBDZznfdiBJ7ALY0idN85H55Pz2X3tvnXfue9PSlecTnMLziz3ww97zgl6</latexit><latexit sha1_base64="M4OHnhlbxja8lJ/D5UF5g/HcIvU=">AAAD/HicnVFNb9QwEJ00fJTw0S0cuUSskFpVu0r2AAipUgUXDggViW0rbXYjJ+tkrTpOsB3UKgr/hBs3xJU/wBXOiH8A/4Kxm64oBYFwFOfNm3kvHk9ScaZ0EHx1VtwLFy9dXr3iXb12/cZab/3mniprmdJxWvJSHiREUc4EHWumOT2oJCVFwul+cvjY5PdfUalYKV7o44pOC5ILlrGUaKTidedelNCciYa+rC3VRpwklGP8UEtWcapjXirVel2dwlNpjAqiFynhzdM2biJNj3TDiczpoMCdibbdjjJJ0iZsm2exbiNVF3HDtsN2dhJzmulJVBFJOKfcz+KoWrAN65pkzVEbsxnZHPyOrjaXsng0Gw3+y0ScNdkqIsnyhZ7GW8NIlKIuEiq9iIr5sl+Dl3fkeXGvHwwDu/zzIOxAH7q1W/a+QARzKCGFGgqgIEAj5kBA4TOBEAKokJtCg5xExGyeQgseamusolhBkD3EPcdo0rECY+OprDrFv3B8JSp9uIuaEuskYvM33+Zr62zYP3k31tOc7Ri/SedVIKthgezfdKeV/6ozvWjI4IHtgWFPlWVMd2nnUttbMSf3f+pKo0OFnMFzzEvEqVWe3rNvNcr2bu6W2Pw3W2lYE6ddbQ3fzSlxwOGv4zwP9kbDMBiGz0f9nUfdqFfhNtyBDZznfdiBJ7ALY0idN85H55Pz2X3tvnXfue9PSlecTnMLziz3ww97zgl6</latexit><latexit sha1_base64="M4OHnhlbxja8lJ/D5UF5g/HcIvU=">AAAD/HicnVFNb9QwEJ00fJTw0S0cuUSskFpVu0r2AAipUgUXDggViW0rbXYjJ+tkrTpOsB3UKgr/hBs3xJU/wBXOiH8A/4Kxm64oBYFwFOfNm3kvHk9ScaZ0EHx1VtwLFy9dXr3iXb12/cZab/3mniprmdJxWvJSHiREUc4EHWumOT2oJCVFwul+cvjY5PdfUalYKV7o44pOC5ILlrGUaKTidedelNCciYa+rC3VRpwklGP8UEtWcapjXirVel2dwlNpjAqiFynhzdM2biJNj3TDiczpoMCdibbdjjJJ0iZsm2exbiNVF3HDtsN2dhJzmulJVBFJOKfcz+KoWrAN65pkzVEbsxnZHPyOrjaXsng0Gw3+y0ScNdkqIsnyhZ7GW8NIlKIuEiq9iIr5sl+Dl3fkeXGvHwwDu/zzIOxAH7q1W/a+QARzKCGFGgqgIEAj5kBA4TOBEAKokJtCg5xExGyeQgseamusolhBkD3EPcdo0rECY+OprDrFv3B8JSp9uIuaEuskYvM33+Zr62zYP3k31tOc7Ri/SedVIKthgezfdKeV/6ozvWjI4IHtgWFPlWVMd2nnUttbMSf3f+pKo0OFnMFzzEvEqVWe3rNvNcr2bu6W2Pw3W2lYE6ddbQ3fzSlxwOGv4zwP9kbDMBiGz0f9nUfdqFfhNtyBDZznfdiBJ7ALY0idN85H55Pz2X3tvnXfue9PSlecTnMLziz3ww97zgl6</latexit>

AnalysisAfter rearrangement:

The gradient:

Llarge-margin =

1

Nt

X

xs2Ss

k f�(xi)� f�(xs) k22 �X

xd2Sd

k f�(xi)� f�(xd) k22

!+ const.

<latexit sha1_base64="qHdwKlUdQsrOIxWR+rRGGL0k4OA=">AAAEFniclVHLbtQwFL1peLTDawpLNhYjpKnQjJLZwAapgg0LhAbBtJXqEjmJM2PVeWA7qFWU/+BP2LFDbNkhFmx5/AXXbkYtHSHAUZLjc+859r03rqTQJgi+eGv+hYuXLq9v9K5cvXb9Rn/z5o4ua5XwWVLKUu3FTHMpCj4zwki+VynO8ljy3fjwsY3vvuFKi7J4aY4rfpCzeSEykTCDVLTpTWnM56Jo+OvaUW2vIzQeb9reBs2ZWSRMNk/bqKGGH5lGMjXnoxy/omhb8pDQTLGkCdvmWWRaKnlmhlTXOeZbcZw1R22kCaGiIC8i3dKKKSYll00W0WohhmfSxNZoldRbp5pXk2hCRmTFP+3s0/+3T8/ZUyXmC7NF7pGTepOy0KYdE1qURZ3HXPUoL9Jlhxw+bV/UHwTjwC2yCsIODKBb07L/GSikUEICNeTAoQCDWAIDjc8+hBBAhdwBNMgpRMLFObTQQ22NWRwzGLKH+J3jbr9jC9xbT+3UCZ4i8VWoJHAXNSXmKcT2NOLitXO27J+8G+dp73aM/7jzypE1sED2b7pl5r/qbC0GMnjgahBYU+UYW13SudSuK/bm5ExVBh0q5CxOMa4QJ0657DNxGu1qt71lLv7DZVrW7pMut4af9pY44PD8OFfBzmQcBuPw+WSw/agb9TrchjswxHneh214AlOYQeK9875637zv/lv/vf/B/3iSuuZ1mlvw2/I//QKpFRNg</latexit><latexit sha1_base64="qHdwKlUdQsrOIxWR+rRGGL0k4OA=">AAAEFniclVHLbtQwFL1peLTDawpLNhYjpKnQjJLZwAapgg0LhAbBtJXqEjmJM2PVeWA7qFWU/+BP2LFDbNkhFmx5/AXXbkYtHSHAUZLjc+859r03rqTQJgi+eGv+hYuXLq9v9K5cvXb9Rn/z5o4ua5XwWVLKUu3FTHMpCj4zwki+VynO8ljy3fjwsY3vvuFKi7J4aY4rfpCzeSEykTCDVLTpTWnM56Jo+OvaUW2vIzQeb9reBs2ZWSRMNk/bqKGGH5lGMjXnoxy/omhb8pDQTLGkCdvmWWRaKnlmhlTXOeZbcZw1R22kCaGiIC8i3dKKKSYll00W0WohhmfSxNZoldRbp5pXk2hCRmTFP+3s0/+3T8/ZUyXmC7NF7pGTepOy0KYdE1qURZ3HXPUoL9Jlhxw+bV/UHwTjwC2yCsIODKBb07L/GSikUEICNeTAoQCDWAIDjc8+hBBAhdwBNMgpRMLFObTQQ22NWRwzGLKH+J3jbr9jC9xbT+3UCZ4i8VWoJHAXNSXmKcT2NOLitXO27J+8G+dp73aM/7jzypE1sED2b7pl5r/qbC0GMnjgahBYU+UYW13SudSuK/bm5ExVBh0q5CxOMa4QJ0657DNxGu1qt71lLv7DZVrW7pMut4af9pY44PD8OFfBzmQcBuPw+WSw/agb9TrchjswxHneh214AlOYQeK9875637zv/lv/vf/B/3iSuuZ1mlvw2/I//QKpFRNg</latexit><latexit sha1_base64="qHdwKlUdQsrOIxWR+rRGGL0k4OA=">AAAEFniclVHLbtQwFL1peLTDawpLNhYjpKnQjJLZwAapgg0LhAbBtJXqEjmJM2PVeWA7qFWU/+BP2LFDbNkhFmx5/AXXbkYtHSHAUZLjc+859r03rqTQJgi+eGv+hYuXLq9v9K5cvXb9Rn/z5o4ua5XwWVLKUu3FTHMpCj4zwki+VynO8ljy3fjwsY3vvuFKi7J4aY4rfpCzeSEykTCDVLTpTWnM56Jo+OvaUW2vIzQeb9reBs2ZWSRMNk/bqKGGH5lGMjXnoxy/omhb8pDQTLGkCdvmWWRaKnlmhlTXOeZbcZw1R22kCaGiIC8i3dKKKSYll00W0WohhmfSxNZoldRbp5pXk2hCRmTFP+3s0/+3T8/ZUyXmC7NF7pGTepOy0KYdE1qURZ3HXPUoL9Jlhxw+bV/UHwTjwC2yCsIODKBb07L/GSikUEICNeTAoQCDWAIDjc8+hBBAhdwBNMgpRMLFObTQQ22NWRwzGLKH+J3jbr9jC9xbT+3UCZ4i8VWoJHAXNSXmKcT2NOLitXO27J+8G+dp73aM/7jzypE1sED2b7pl5r/qbC0GMnjgahBYU+UYW13SudSuK/bm5ExVBh0q5CxOMa4QJ0657DNxGu1qt71lLv7DZVrW7pMut4af9pY44PD8OFfBzmQcBuPw+WSw/agb9TrchjswxHneh214AlOYQeK9875637zv/lv/vf/B/3iSuuZ1mlvw2/I//QKpFRNg</latexit><latexit sha1_base64="qHdwKlUdQsrOIxWR+rRGGL0k4OA=">AAAEFniclVHLbtQwFL1peLTDawpLNhYjpKnQjJLZwAapgg0LhAbBtJXqEjmJM2PVeWA7qFWU/+BP2LFDbNkhFmx5/AXXbkYtHSHAUZLjc+859r03rqTQJgi+eGv+hYuXLq9v9K5cvXb9Rn/z5o4ua5XwWVLKUu3FTHMpCj4zwki+VynO8ljy3fjwsY3vvuFKi7J4aY4rfpCzeSEykTCDVLTpTWnM56Jo+OvaUW2vIzQeb9reBs2ZWSRMNk/bqKGGH5lGMjXnoxy/omhb8pDQTLGkCdvmWWRaKnlmhlTXOeZbcZw1R22kCaGiIC8i3dKKKSYll00W0WohhmfSxNZoldRbp5pXk2hCRmTFP+3s0/+3T8/ZUyXmC7NF7pGTepOy0KYdE1qURZ3HXPUoL9Jlhxw+bV/UHwTjwC2yCsIODKBb07L/GSikUEICNeTAoQCDWAIDjc8+hBBAhdwBNMgpRMLFObTQQ22NWRwzGLKH+J3jbr9jC9xbT+3UCZ4i8VWoJHAXNSXmKcT2NOLitXO27J+8G+dp73aM/7jzypE1sED2b7pl5r/qbC0GMnjgahBYU+UYW13SudSuK/bm5ExVBh0q5CxOMa4QJ0657DNxGu1qt71lLv7DZVrW7pMut4af9pY44PD8OFfBzmQcBuPw+WSw/agb9TrchjswxHneh214AlOYQeK9875637zv/lv/vf/B/3iSuuZ1mlvw2/I//QKpFRNg</latexit>

@Llarge-margin

@f�(xi)=

2

Nt

X

xs2Ss

(f�(xi)� f�(xs))�X

xd2Sd

(f�(xi)� f�(xd))

!

= �2|Ss|Nt

1

|Ss|X

xs2Ss

f�(xs)� f�(xi)

!� 2|Sd|

Nt

f�(xi)�

1

|Sd|X

xd2Sd

f�(xd)

!

= � 2|Ss|Nt

(cs � f�(xi))| {z }

pull towards its own class

� 2|Sd|Nt

(f�(xi)� cd)| {z }

push away from other classes

.

<latexit sha1_base64="KKiGFv7+9REOYQvnl8Jmcqj5MVY=">AAAGB3icjVPNbtNAEJ4SAiX8tXDksqICJYdWSS8gJKQCFw4IFUF/pG5lre1Nsur6h901beXuA/Am3LghrrwA18IbwFswu3aaQkxbR3HG38x8832zTphLoU2//3PuUuty+8rV+Wud6zdu3rq9sHhnU2eFivhGlMlMbYdMcylSvmGEkXw7V5wloeRb4d4Ll9/6wJUWWfrOHOZ8N2GjVAxFxAxCwWLrGQ35SKQlk2KU2g6hQ8WikuZMGcEkTZgZR0yWr2xQUsMPTCmZGvHlBO8itdaelJJhQPOx6PqWcFge2ED0LHn4dMK5asvXgbFU8qHpUl0kyDit1YRQkZK3gbbdJqblWVD3emSZzDDFFU98YZ6416NKjMamR2iapUUSckUo7aB2x1+JP0JlR5WDDkGtlQuXG5RV7ixLTeIblIiJEBxxenI8M7nR2lRO3CDnZC9NGzhrAUUacxUiNS/PXEbNF6G5c7zZoHqX8kJKYrJ9pmJNhNEk209JJJnWttpA4+iLbmOqJ24arceE7bNDMlRZQjIzRs9+NNd2BXlZyGXJ3z8ZKRYLnho73U2H8jSe/GM6wcJSf6XvLzIbDOpgCeprPVs4BgoxZBBBAQlwSMFgLIGBxs8ODKAPOWK7UCKmMBI+z8FCB3sLrOJYwRDdw/sIn3ZqNMVnx6l9d4RTJH4VdhJ4gD0Z1imM3TTi84Vnduj/uEvP6bQd4m9YcyWIGhgjel7fpPKifc6LgSE89h4Eeso94txFNUvht+KUk1OuDDLkiLk4xrzCOPKdkz0T36O9d7db5vO/fKVD3XNU1xbw26nEAx78e5yzwebqyqC/MnizurT2vD7qebgH96GL5/kI1uAlrMMGRK1Pre+t49aP9sf25/aX9teq9NJc3XMX/rra3/4A/X7G0A==</latexit><latexit sha1_base64="KKiGFv7+9REOYQvnl8Jmcqj5MVY=">AAAGB3icjVPNbtNAEJ4SAiX8tXDksqICJYdWSS8gJKQCFw4IFUF/pG5lre1Nsur6h901beXuA/Am3LghrrwA18IbwFswu3aaQkxbR3HG38x8832zTphLoU2//3PuUuty+8rV+Wud6zdu3rq9sHhnU2eFivhGlMlMbYdMcylSvmGEkXw7V5wloeRb4d4Ll9/6wJUWWfrOHOZ8N2GjVAxFxAxCwWLrGQ35SKQlk2KU2g6hQ8WikuZMGcEkTZgZR0yWr2xQUsMPTCmZGvHlBO8itdaelJJhQPOx6PqWcFge2ED0LHn4dMK5asvXgbFU8qHpUl0kyDit1YRQkZK3gbbdJqblWVD3emSZzDDFFU98YZ6416NKjMamR2iapUUSckUo7aB2x1+JP0JlR5WDDkGtlQuXG5RV7ixLTeIblIiJEBxxenI8M7nR2lRO3CDnZC9NGzhrAUUacxUiNS/PXEbNF6G5c7zZoHqX8kJKYrJ9pmJNhNEk209JJJnWttpA4+iLbmOqJ24arceE7bNDMlRZQjIzRs9+NNd2BXlZyGXJ3z8ZKRYLnho73U2H8jSe/GM6wcJSf6XvLzIbDOpgCeprPVs4BgoxZBBBAQlwSMFgLIGBxs8ODKAPOWK7UCKmMBI+z8FCB3sLrOJYwRDdw/sIn3ZqNMVnx6l9d4RTJH4VdhJ4gD0Z1imM3TTi84Vnduj/uEvP6bQd4m9YcyWIGhgjel7fpPKifc6LgSE89h4Eeso94txFNUvht+KUk1OuDDLkiLk4xrzCOPKdkz0T36O9d7db5vO/fKVD3XNU1xbw26nEAx78e5yzwebqyqC/MnizurT2vD7qebgH96GL5/kI1uAlrMMGRK1Pre+t49aP9sf25/aX9teq9NJc3XMX/rra3/4A/X7G0A==</latexit><latexit sha1_base64="KKiGFv7+9REOYQvnl8Jmcqj5MVY=">AAAGB3icjVPNbtNAEJ4SAiX8tXDksqICJYdWSS8gJKQCFw4IFUF/pG5lre1Nsur6h901beXuA/Am3LghrrwA18IbwFswu3aaQkxbR3HG38x8832zTphLoU2//3PuUuty+8rV+Wud6zdu3rq9sHhnU2eFivhGlMlMbYdMcylSvmGEkXw7V5wloeRb4d4Ll9/6wJUWWfrOHOZ8N2GjVAxFxAxCwWLrGQ35SKQlk2KU2g6hQ8WikuZMGcEkTZgZR0yWr2xQUsMPTCmZGvHlBO8itdaelJJhQPOx6PqWcFge2ED0LHn4dMK5asvXgbFU8qHpUl0kyDit1YRQkZK3gbbdJqblWVD3emSZzDDFFU98YZ6416NKjMamR2iapUUSckUo7aB2x1+JP0JlR5WDDkGtlQuXG5RV7ixLTeIblIiJEBxxenI8M7nR2lRO3CDnZC9NGzhrAUUacxUiNS/PXEbNF6G5c7zZoHqX8kJKYrJ9pmJNhNEk209JJJnWttpA4+iLbmOqJ24arceE7bNDMlRZQjIzRs9+NNd2BXlZyGXJ3z8ZKRYLnho73U2H8jSe/GM6wcJSf6XvLzIbDOpgCeprPVs4BgoxZBBBAQlwSMFgLIGBxs8ODKAPOWK7UCKmMBI+z8FCB3sLrOJYwRDdw/sIn3ZqNMVnx6l9d4RTJH4VdhJ4gD0Z1imM3TTi84Vnduj/uEvP6bQd4m9YcyWIGhgjel7fpPKifc6LgSE89h4Eeso94txFNUvht+KUk1OuDDLkiLk4xrzCOPKdkz0T36O9d7db5vO/fKVD3XNU1xbw26nEAx78e5yzwebqyqC/MnizurT2vD7qebgH96GL5/kI1uAlrMMGRK1Pre+t49aP9sf25/aX9teq9NJc3XMX/rra3/4A/X7G0A==</latexit><latexit sha1_base64="KKiGFv7+9REOYQvnl8Jmcqj5MVY=">AAAGB3icjVPNbtNAEJ4SAiX8tXDksqICJYdWSS8gJKQCFw4IFUF/pG5lre1Nsur6h901beXuA/Am3LghrrwA18IbwFswu3aaQkxbR3HG38x8832zTphLoU2//3PuUuty+8rV+Wud6zdu3rq9sHhnU2eFivhGlMlMbYdMcylSvmGEkXw7V5wloeRb4d4Ll9/6wJUWWfrOHOZ8N2GjVAxFxAxCwWLrGQ35SKQlk2KU2g6hQ8WikuZMGcEkTZgZR0yWr2xQUsMPTCmZGvHlBO8itdaelJJhQPOx6PqWcFge2ED0LHn4dMK5asvXgbFU8qHpUl0kyDit1YRQkZK3gbbdJqblWVD3emSZzDDFFU98YZ6416NKjMamR2iapUUSckUo7aB2x1+JP0JlR5WDDkGtlQuXG5RV7ixLTeIblIiJEBxxenI8M7nR2lRO3CDnZC9NGzhrAUUacxUiNS/PXEbNF6G5c7zZoHqX8kJKYrJ9pmJNhNEk209JJJnWttpA4+iLbmOqJ24arceE7bNDMlRZQjIzRs9+NNd2BXlZyGXJ3z8ZKRYLnho73U2H8jSe/GM6wcJSf6XvLzIbDOpgCeprPVs4BgoxZBBBAQlwSMFgLIGBxs8ODKAPOWK7UCKmMBI+z8FCB3sLrOJYwRDdw/sIn3ZqNMVnx6l9d4RTJH4VdhJ4gD0Z1imM3TTi84Vnduj/uEvP6bQd4m9YcyWIGhgjel7fpPKifc6LgSE89h4Eeso94txFNUvht+KUk1OuDDLkiLk4xrzCOPKdkz0T36O9d7db5vO/fKVD3XNU1xbw26nEAx78e5yzwebqyqC/MnizurT2vD7qebgH96GL5/kI1uAlrMMGRK1Pre+t49aP9sf25/aX9teq9NJc3XMX/rra3/4A/X7G0A==</latexit>

Case study- We implement and compare several of other large marginmethods for few-shot learning.- Our framework is simple, efficient, and can be applied toimprove existing and new meta-learning methods with very littleoverhead.

Features

- Graph Neural Network (GNN)- Prototypical Network (PN)

The University of Hong Kong1, The Hong Kong Polytechnic University2

Yong Wang1, Xiao-Ming Wu2, Qimai Li2, Jiatao Gu1, Wangmeng Xiang2, Lei Zhang2, Victor O.K. Li1

Amortized Bayesian Meta-Learning Sachin Ravi & Alex Beatson

Department of Computer Science, Princeton University

‣ Lot of progress in few-shot learning but under controlled settings

‣ In real world, relationship between training and testing tasks can be tenuous

‣ Task-specific predictive uncertainty is crucial

‣ We present gradient-based meta-learning method for computing task-specific approximate posterior

‣ Show that method displays good predictive uncertainty on contextual-bandit and few-shot learning tasks

The effects of negative adaptation in Model-Agnostic Meta-Learning

Tristan Deleu, Yoshua Bengio

• The advantage of meta-learning is well-founded under the assumption that the adaptation phase does improve the performance of the model on the task of interest

• Optimization: maximize the performance after adaptation, performance improvement is not explicitly enforced

• We show empirically that performancecan decrease after adaptation in MAML.We call this negative adaptation

• How to fix this issue? Ideas fromSafe Reinforcement Learning

min✓

ET ⇠p(T )[L(✓0T ;D0T )]

Audrey G. Chung, Paul Fieguth, Alexander Wong

Evolvability ES: Scalable Evolutionary Meta-Learning

● Evolvability ES is a meta-learning algorithm inspired by Evolution Strategies [1]

● Surprisingly, Evolvability ES finds parameters such that at test time, random perturbations result in diverse behaviors

● In a simulated Ant locomotion domain, adding Gaussian noise to the parameters results in policies which move in many different directions

By Alexander Gajewski, Jeff Clune, Kenneth O. Stanley, and Joel Lehman

[1] Salimans et al., Evolution Strategies as a Scalable Alternative to Reinforcement Learning, 2017.

Consolidating the Meta-Learning ZooA Unifying Perspective as Posterior Predictive Inference

► Novel: Probabilistic, amortized, multi-task, meta-learning framework.

►Meta-learning: Learns how to learn a classifier or regressor for each new task.

► Unifies: MAML, Meta-LSTM, Prototypical networks, and Conditional Neural Processes are special cases.

► State of the art: Leading classification accuracy on 5 of 6 Omniglot & miniImageNet tasks.

► Efficient: Test-time requires only forward passes, no gradient steps are needed.

► Versatile: Robust classification accuracy as shot and way are varied at test-time.

Jonathan Gordon1, John Bronskill1, Matthias Bauer1,2, Sebastian Nowozin3, Richard E. Turner1,3

1University of Cambridge, 2MPI for Intelligent Systems, Tübingen, 3Microsoft Research

► High quality 1-shot view reconstruction:

Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RLAnusha Nagabandi, Chelsea Finn, Sergey Levine

Can we use meta-learning for effective online learning?

Our method can: - Reason about non-stationary

latent distributions over tasks. - Recall past tasks