-
[Conceptual Background] Rethinking on Weight Decay
In this post, I intend to reflect on Weight Decay, which is an almost essential element in Deep Learning. Existing posts regarding WD mostly explain the phenomenon itself. So, in this post, I intend to examine in what context WD is actually used, and in what sense it is being used recently.
-
[Information Theory] Asymptotic Equipartition Property (AEP)
Recently, while studying AI, I naturally began to think a lot from the perspective of Information Theory. In that process, I took another deep look at the theory itself, and since I had not posted much on my blog for a while, I thought this would be a good opportunity to organize and share what I have studied, using the textbook "Elements of Information Theory" as the main reference. I plan to make it as easy to understand as possible, with explanations centered on intuitive examples and accompanied by formulas and proofs only where absolutely necessary.
-
[Coding] Super Easy Guide to Applying PyTorch DDP
This is my first coding-related post, and the topic is DDP. Recently, as model capacities have grown, using multiple GPUs has become essential. Consequently, knowing how to use DDP effectively has become very important. Therefore, in this post, I will share how to apply DDP. I will cut to the chase on the general mechanics and focus simply and clearly on the arguments. (I will show the method I personally use!)
-
[Theoretical Background] Convex Optimization 2
I’m back with the second post related to Optimization. Last time, we examined how Gradient Descent converges when the function is $L$-Lipschitz. This time, let's examine what happens when stronger constraints or assumptions are applied. Without further ado, let's jump right in.
-
[Theoretical Background] Convex Optimization 1
This year, I took a class on Optimization for the first time. In the Deep Learning field, Optimization is a foundational discipline for mathematically verifying whether convergence occurs and, if so, how quickly it occurs. Although I have previously posted about Meta-Learning and Generative Models, and announced that I would post about Foundation Models in the future, I also thought it would be good to share some concepts based on mathematics. So, I plan to post mainly about what I have learned in my graduate school classes. Since I have organized these materials, I plan to post them periodically without making excuses.