Jack (in SF) Langerman

@jacklangerman.bsky.social

67 Followers 53 Following

LoRA+: Efficient Low Rank Adaptation of Large Models Soufiane Hayou, Nikhil Ghosh, Bin Yu In this paper, we show that Low Rank Adaptation (LoRA) as originally introduced in Hu et al. (2021) leads to suboptimal finetuning of models with large width (embedding dimension). This is due to the fact that adapter matrices A and B in LoRA are updated with the same learning rate. Using scaling arguments for large width networks, we demonstrate that using the same learning rate for A and B does not allow efficient feature learning. We then show that this suboptimality of LoRA can be corrected simply by setting different learning rates for the LoRA adapter matrices A and B with a well-chosen ratio. We call this proposed algorithm LoRA+. In our extensive experiments, LoRA+ improves performance (1-2 % improvements) and finetuning speed (up to ∼ 2X SpeedUp), at the same computational cost as LoRA.
:_( operation not implemented on MPS woe is me...
If you'd like to see a concrete example of REDACTED could be implemented or compared to REDACTED (e.g., in PyTorch), let me know! I can walk you through code or provide further insights. 🚀
Andrej Karpathy @karpathy Al video generation today. When I was back in school, the story of the field of computer graphics (and physically based rendering etc.) was that we will carefully study and model all the object/scene geometry, physics rendering etc., and after 1000 PhDs and 50 SIGGRAPHs get results like this. That a Transformers can shortcut all of that at this high of fidelity by training on a dataset of videos... Agrim Gupta @agrimgupta92 . 2h "A pair of hands skillfully slicing a ripe tomato on a wooden cutting board" #veo
[138]: generate_with_vector( "Who am I speaking to? Please describe yourself, including any physical details. vec("python") * 0.5, # show baseline-True, temperature=l, <|begin_of_text|> system Cutting Knowledge Date: December 2023 Today Date: 26 Jul 2024 user Who am I speaking to? Please describe yourself, including any physical details assistant **1 am Python, a Python programmer.'* I'm a Python enthusiast with a passion for coding, not a snake. I'm here to assist with Python programming, specifically with the Python programming language. **Physical Details:** I'm a programmer, not a reptile. I have a Python logo on my gear, but I don't have scales or a slithery body. l'm a programmer who loves Python and is here to help with your Python-related queries. **Language Support:** I'll provide Python code in the following format: python # Python code here
Figure 1: Causal Traces compute the causal effect of neuron activations by running the network twice: (a) once normally, and (b) once where we corrupt the subject token and then (c) restore selected internal activations to their clean value. (d) Some sets of activations cause the output to return to the original prediction; the light blue path shows an example of information flow. The causal impact on output probability is mapped for the effect of (e) each hidden state on the prediction, (f) only MLP activations, and (g) only attention activations
Figure 2. Plug-and-play Diffusion Features. (a) Our framework takes as input a guidance image and a text prompt describing the desired translation; the guidance image is inverted to initial noise xG T, which is then progressively denoised using DDIM sampling. During this process, we extract (f l t, ql t, kl t) – spatial features from the decoder layers and their self-attention, as illustrated in (b). To generate our text-guided translated image, we fix x∗ T = xG T and inject the guidance features (f l t, ql t, kl t) at certain layers, as discussed in Sec. 4.
Figure 1: Schematic of the Activation Addition (ActAdd) method.= natural language text; •= vectors of activations just before a specified layer. In this example, the output is heavily biased towards discussing weddings, regardless of the topic of the user prompt. (See Algorithm 1 for the method’s parameters: intervention strength, intervention layer, and sequence alignment.)
Est. mag 6.9 earthquake 240.6 miles away December 5, 2024, 10:44 AM Source: ShakeAlert@ OREGON