Awesome List Updates on Oct 07, 2023

7 awesome lists updated today.

🏠 Home · 🔍 Search · 🔥 Feed · 📮 Subscribe · ❤️ Sponsor

1. Awesome Azure Openai Llm

Finetuning / PEFT: Parameter-Efficient Fine-Tuning (Youtube) [24 Apr 2023]

• Efficient Streaming Language Models with Attention Sinks: [cnt] 1. StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence length without any fine-tuning. 2. We neither expand the LLMs' context window nor enhance their long-term memory. git (⭐6.4k) [29 Sep 2023]

  Expand: StreamingLLM
  • Key-Value (KV) cache is an important component in the StreamingLLM framework.
  1. Window Attention: Only the most recent Key and Value states (KVs) are cached. This approach fails when the text length surpasses the cache size.
  2. Sliding Attention /w Re-computation: Rebuilds the Key-Value (KV) states from the recent tokens for each new token. Evicts the oldest part of the cache.
  3. StreamingLLM: One of the techniques used is to add a placeholder token (yellow-colored) as a dedicated attention sink during pre-training. This attention sink attracts the model’s attention and helps it generalize to longer sequences. Outperforms the sliding window with re-computation baseline by up to a remarkable 22.2× speedup.

2. Awesome Qubes OS

Optics and Extra Info / Unikernel-like

• Qubes OS Summit 2023 - Day 1

3. Awesome List

Decentralized Systems

• Mastodon (⭐24) - Open source decentralized microblogging network.

4. Web Development Resources

Coding Challenge Platforms:

• Website: https://divize.io/challenges

Animation Libraries:

• Website: Motion One

5. Awesome Datascience

Colleges

• BS in Data Science & Applications

Comparison / Unsupervised Learning

• Neural Networks

Comparison / Deep Learning architectures

• Multilayer Perceptron

• Convolutional Neural Network (CNN)

• Recurrent Neural Network (RNN)

• Boltzmann Machines

• Autoencoder

• Generative Adversarial Network (GAN)

• Self-Organized Maps

• Transformer

• Conditional Random Field (CRF)

6. Awesome Pinned Gists

External Services

• osu-box (⭐7) - Update a pinned gist to display your osu! stats.

7. Free Programming Books (English, By Subjects)

Algorithms & Data Structures

• Algorithmic Thinking - Donglai Fu

Open Source Ecosystem

• 500 lines or less (⭐29k) - Build from Source Code

Quantum Computing

• Introduction to Quantum Information - Stephen M. Barnett (PDF)

• Prev: Oct 08, 2023
• Next: Oct 06, 2023