-
[Conceptual Background] Rethinking on Weight Decay
In this post, I intend to reflect on Weight Decay, which is an almost essential element in Deep Learning. Existing posts regarding WD mostly explain the phenomenon itself. So, in this post, I intend to examine in what context WD is actually used, and in what sense it is being... [Read More] -
[Information Theory] Asymptotic Equipartition Property (AEP)
Recently, while studying AI, I naturally began to think a lot from the perspective of Information Theory. In that process, I took a deep look into the theory itself again, and since my blog content had been sparse for a while, I thought this would be a good opportunity to... [Read More] -
[Coding] Super Easy Guide to Applying PyTorch DDP
This is my first coding-related post. The topic is DDP. Recently, as model capacities have grown, using multiple GPUs has become essential. Consequently, utilizing DDP effectively has become very important. Therefore, in this post, I will share how to apply DDP. I will cut to the chase regarding the general... [Read More] -
[Theoretical Background] Convex Optimization 2
I’m back with the second post related to optimization. Last time, we examined how it expands and converges when the function is L-lipschitz. This time, let's examine what happens when stronger constraints / assumptions are applied. Then, without further ado, let's jump right in. $\star$ Recap (Previous Post) In the... [Read More] -
[Theoretical Background] Convex Optimization 1
This year, I took a class involving the concept of Optimization for the first time. In dealing with the Deep Learning field, it was accepted like a foundational discipline to mathematically verify whether convergence happens, and if so, how fast it happens. While the Meta Learning and Generative Models I... [Read More] -
[Conceptual Background] Meta Learning (2) - Approaches
In the previous post, I briefly explained the context of how meta-learning emerged and the basic concept of few-shot learning to help understand meta-learning. So, in this post, I intend to explain meta-learning approaches. I plan to summarize what the pioneer papers are and what points each paper wants to... [Read More] -
[Conceptual Background] Meta Learning (1) - Few-Shot Learning
I would like to introduce the basic concepts of Meta Learning, a topic I have been putting off for a while. Over the past few years, Meta Learning has been a hot topic, consistently making its name as a Main Keyword in Top Tier Conferences. Especially considering the circulation speed... [Read More] -
[Paper Review] Sharp-MAML
Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning For my first blog post, I would like to write about Sharp-MAML, which combines SAM, a hot topic in the generalization field these days (since 2020), and MAML, the pioneer of Meta Learning algorithms. After briefly introducing the SAM algorithm and MAML, I will explain... [Read More]