-
[Conceptual Background] Rethinking on Weight Decay
In this post, I intend to reflect on Weight Decay, which is an almost essential element in Deep Learning. Existing posts regarding WD mostly explain the phenomenon itself. So, in this post, I intend to examine in what context WD is actually used, and in what sense it is being used recently.
-
[Information Theory] Asymptotic Equipartition Property (AEP)
Recently, while studying AI, I naturally began to think a lot from the perspective of Information Theory. In that process, I took a deep look into the theory itself again, and since my blog content had been sparse for a while, I thought this would be a good opportunity to organize and post what I've studied, centering on the textbook "Elements of Information Theory". I plan to organize it to be as easy to understand as possible, with explanations centered on intuitive examples, accompanied by formulas and proofs where absolutely necessary.
-
[Coding] Super Easy Guide to Applying PyTorch DDP
This is my first coding-related post. The topic is DDP. Recently, as model capacities have grown, using multiple GPUs has become essential. Consequently, utilizing DDP effectively has become very important. Therefore, in this post, I will share how to apply DDP. I will cut to the chase regarding the general mechanics and focus simply and clearly on the arguments. (I will proceed with the method I personally use!)
-
[Theoretical Background] Convex Optimization 2
I’m back with the second post related to optimization. Last time, we examined how it expands and converges when the function is L-lipschitz. This time, let's examine what happens when stronger constraints / assumptions are applied. Then, without further ado, let's jump right in.
-
[Theoretical Background] Convex Optimization 1
This year, I took a class involving the concept of Optimization for the first time. In dealing with the Deep Learning field, it was accepted like a foundational discipline to mathematically verify whether convergence happens, and if so, how fast it happens. While the Meta Learning and Generative Models I posted previously, and the Foundation Models I announced I would post in the future, are good, I thought it would also be good to upload concepts based on mathematics. So, I plan to try posting mainly about the content I learned during my graduate school classes. Since I have organized what I learned regarding these contents, I plan to post them periodically without making excuses.