Why Models Are AI’s Next Training Dataset with Damian Borth - #772

One of the biggest questions facing AI today is how foundation models keep improving as high quality training data becomes harder to find. Some researchers are betting on synthetic data, others on inference, time, reasoning. Today's guest has his chips on something very different. Every trained model represents thousands or even millions of GPU hours spent discovering what works. Instead of treating those weights just as the end of the training process, what if they're also the beginning of the next one? Damian Borth, professor of AI and machine learning at the University of St. Gallen, sees trained models themselves as data. Data that can be learned from, analyzed, and even used to generate entirely new models. When I asked him to explain the idea behind weight space learning, here's where he started.

0:00

So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there Much, much better. Thinking about that, that you can treat the weights as a input modality gives you suddenly this opportunity of can we be much, much faster in creating new weights for particular tasks, or can we be much more precise in analyzing weights? When somebody gives me a new network that I'm not knowledgeable about and I never saw before.

0:47

I'm Sam Charrington, TWIML AI podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

1:23

We started like in 2000 2021, the work on what we call weight space learning. And weight space learning is a quite interesting way at looking at machine learning in general. That's currently the major topic. We also do a little bit of work in remote sensing and then representation learning on tabular data. We're now walking and combining everything together to focus more on weight space learning, which I think is a really interesting way forward, solves a couple of problems, problems that the community currently encounters and started from a very esoteric idea to something that works surprisingly well.

1:46

You know, we think about weights as the product of training a model and, you know, we get some utility out of them. We may use them for things like explainability or manipulate them when we're quantizing or something like that. But the idea seems to be that, you know, there's so much more that we can learn from these weights.

2:30

Exactly. So if you think about machine learning, machine learning has this idea of you have data and some output in classical supervised machine learning data and some predictions, and you Train neural network in between to mimic the data set, mimic the distribution of the data set. And the outcome during this very expensive training procedure is, is a set of weights, a configuration of parameters that define the neural network, like the DNA of the neural network. This is classical machine learning, supervised, unsupervised, self supervised. That fuels a lot of innovation over the last 10 years. And with GEN moved to the next stage, if you look at what happened over the last couple of years, more and more and more of those models have been published publicly, are online accessible at, you know, repositories like hugging face or GitHub. So we basically thought about this very simple idea. What happens actually if we take the weights of trained neural networks as the input to train a neural network to understand these weights that we have out there much, much better. So to take another analogy, language models, you take a big model, you train this on every single sentence on the Internet. At the end you have a language model able to analyze language and to generate language. You can take the same idea for pixels. You take a big model, you train on all the pixels on the Internet and you can analyze pixels and you can generate pixels. We do the same idea on all the weights of trained neural networks. So we can analyze weights of neural networks and we can generate weights of neural networks. As straightforward as it is, obviously there's a little bit more into the details, but thinking about that, that you can treat the weights as a input modality gives you suddenly this opportunity of thinking about, okay, what would be language translation with more like neural network models, right? What would be generation of words and tokens that are worse in generation of tokens that are weights? And can we be much, much faster in creating new weights for particular tasks or can we be much more precise in analyzing weights? When somebody gives me a new network that I'm not knowledgeable about and I never saw before, and then, you know, you have this new entire world, this, you know, empty space of things you can do with weights that, you know, you kind of carry into the community and hope that there's somebody listening and continuing and, you know, building up a community which happened over the last two, three years, which is very exciting because there are more people about that. And yeah, weights are exciting not only as the output of learning, but as the input for learning.

2:51

You mentioned you started this effort in 2021. Where did you start from? And then we'll work our way towards like where we are now with weights based learning.

5:41

Originally this idea came 2020 and we got the first paper published 21. And the idea was very simple. Can we fingerprint a neural network or version a neural network like we can do with software, right? In software you have, you can do a diff, right? You have a 1 million lines of code. Somebody changes something and then, you know, you do, you do a diff, you know exactly where the difference is. So can we do this with neural networks? Problem with neural networks is if you do one update of weights during training, every weight is a little bit different. So there's not much you can extract from this, right?

5:51

They're fairly unstable locally.

6:25

If everything is different, nothing is different, right? So we were thinking about, can we find a space where these neural networks, the weights, the models of those neural networks are a little bit compressed and more understandable. And we started to think about that. And in parallel there was this really amazing work. The first work I from Thomas Untertinger and Daniel Kaisers and colleagues from Google Zurich, and they developed a paper that used weights as input, you know, did some statistical features, handcrafted features, and then predicted the accuracy of those weights. And another paper by Ellison that predicted the generalization gap. So people started to use weights to extract information and they were all handcrafted features. So I was thinking about handcrafted features that sold traditional machine learning. So why not end to end learning? So then we developed our idea of auto encoding sequences of weights into a lower dimensional space and then reconstructing it. And if we're able to do this from a population of neural networks, then we can maybe learn a lower dimension manifold that populates actually where the neural networks populate that manifold and maybe just manifold encodes information about accuracy. What training data was used, what training fraction learning rate and all this latent generating factors. And we started working on this and the first paper was really like, oh, well, it works. We can compress neural networks like very small, tiny neural networks, toy examples, right? Embarrassingly small, like thousands, ten thousands of parameters. But it was working. And we could predict the accuracies and, and it was very nice to see that. And then, you know, we got published.

6:27

What exactly were you able to predict?

8:18

So we, we took, you know, an auto encoder, we have an encoder and decoder. We learned the auto encoder with or trained the automotive reconstruction loss, a contrastive loss in the middle. And then we take, we took only the encoder and unknown neural networks that we encoded into the latent space. And these embeddings we put into a simple regression, like a linear regression head to predict the accuracy so you give me a neural network. I never saw this neural network. And the idea was, can I predict the accuracy of this neural network? Can I test the neural network without the use of test data? Obviously this worked on this really small neural networks only in a homogeneous. We call this a model zoo. A population of models. So they were all trained on the same data, same architecture. First, you know, small steps towards this goal. But we could extract this information about the accuracy, the performance, the generalization gap, what kind of activation function was used. And if you would plot this latent spaces, you would see different initializations and how they evolve. Because we had, you know, 50 epochs per model training and 1,000 models. So little trajectories were visible. So there was structure. And they all organized around a latent space.

8:21

The metadata around performance in accurate. These are things that you had from the base models. And so it was supervised in that sense.

9:40

Exactly. So we needed to be able to train this autoencoder some population of neural networks, model zoo. And at that time, you know, laboratory trained under known condition, fully transparent. We knew which model in which epoch had which accuracy. We had obviously trained test and validation splits. The alto encoder was trained on 600 neural networks and tested on 300 others. And then we could compare. We predicted 90% accuracy on Fashion MNIST and the model had 92. And then we were R squared. And we outperformed the work from Thomas Huntertinga and Daniel Kaiser. So we were happy outperformed the original weightspace. You know, that story. It worked. We got the paper and people. I'm very thankful to the reviewers, they were kind of telling us that small. But it's interesting and we're able to publish because that was kind of the ignition this amazing journey that we had over the last four years.

9:49

So I can imagine, you know, lots of different directions, including scaling up the models, trying to get more insights out of the space, like what was next.

10:50

The thing is, for this type of research, everything was very obvious. It was like lying out. And you just needed to do it. I mean, it's. I never had this before. Right. So you have an auto encoder. You take the encoder so it can predict discriminative downstream tasks like what's the accuracy, what's generalization gap. But we have the other thing called the decoder. So can we sample from this space to generate neural networks? It's obvious, right? We didn't have space in the first paper, so we needed one more year to have a 22 paper published on Generating neural networks. Hopefully those neural networks were then better than standard initializations. They were not as good as final or fully trained neural networks. So there was some trouble that we had which was really, really interesting. But we, we trained this out encoder. The mean split error was super low. We took the neural networks, we moved them in the forward pass, we reconstructed as said, the loss is very low. We plug the weights back to the neural network. Totally screwed up the entire neural network. We're like, yeah, it was like, it was really like that means low and obviously now obviously right. A mean spreader is an average. So we're, we very good at reconstructing the average weights. But the little things that make the difference of having this function working or not, they were so important. And there is an analogy to pixels and images. Like when you had generative models for images, the images were always blurry. So people kind of tried hard and you know, taming transformers for high resolution images changed the mean squared error to perception laws and you know, did some additional things on quantizing and the gan. So we knew that reconstruction may be the wrong loss. And we tried to normalize and play around with the losses and you know, thought about the behavior loss. And then we were able to make those models a little bit better. But there's still a little bit of delta that is missing. We kind of, we generate at that time blurry weights, right? Low frequency information, high frequency information missing. And we know, we were happy because again, you know, people were kind to us and said like it's toy examples. But you know, it's interesting we generate that the numbers are good. But then I said, okay, we cannot be three times lucky. So we have to work really hard to scale that up. Right? I mean, you know, you're lucky twice, but you know, three times, you know, your karma is gone for the next years. So we then really and very thankful to, you know, Constantin, who was part of that initial phase. And he really worked hard. And we had this idea of instead of reconstructing the entire model, think about the model parameters as a sequence that you window and then reconstruct the windows. And therefore we would kind of de attach the sequence left of the original model to the autoencoder one. And this was interesting because suddenly we could go to Resnet and beyond. And this led then to the work 2024 in collaboration with Michael Mahoney from UC Berkeley. He actually said in one of the discussions like Damien, what you're doing is really interesting, but useless. And say like, yeah, sure, a Lot of research starts like that. So. And then, you know, because he said that I asked him that you have to help to scale that up. Right. So I caught him and he was then one of the co authors on the paper. Also the previous work was with a lot of collaboration, you know, Xavi Nero and Boris Knasny, they were part of that. Because in the beginning the idea was a little bit, as I mentioned, esoteric. So we were wondering, like, what are other people thinking about? So we very early involved a lot of people from the community to double check if we are the crazy ones or if this idea is, you know, to at least a particular limit, meaningful. So we Then in 24, we scaled up to larger networks and other people got interested and we were at the conference and we met. Oh, there exist other people that are doing similar things. A lot of work from Technion, you know, Hagai Maron, Gel Chetnik, Yedit. And then people like, you know, Eliahu was actually a funny thing. There was one student that came to our poster and he has this kind of, you know, badge. And at the badge at the bottom you have always the university written. And instead of the university, he had like weights at the New Modality, Elaho Hovitz. And I was like, oh, that's exactly what I'm thinking. And then a collaboration started, like a commute. And you know, we didn't.

11:00

We.

15:18

We recognized the other people and we said, like, why not doing a workshop? And then, you know, one led to another. And then, you know, that was last year. I saw 2024, we had proposal 2025, we had then the first workshop. And then, you know, the other people recognize there are other areas that are very important. One of the tricky things with weight spaces is when you have a neural network and you have two layers, let's say for simplicity, fully connected, you can change the position of the neurons and it actually changes the sequence of weights because, you know, the order of weights, but the function is the same. So it's, you know, there's little permutation symmetries and other, you know, things in the weight space that do not change the underlying function. So we had it already in the first paper, some augmentation. But there are a lot of people that are very, you know, very specialized on that and much more experienced, much, you know, more theoretical on that.

15:18

So augmentation in the sense of like applying these identity transformations to your training, your input models and using them to increase generalization and.

16:17

Exactly, yeah. Very simply, in our work in 21, we had a contrastive loss. And to build a contrast you need to augment. So I mean, it's simple to flip an image or rotate an image. But, you know, what's the counterpart in weight spaces, right? I mean, and you can do the permutations and you can do other people that invented, you know, scale augmentations and other things. So the field was exciting because you saw things happening in nlp, you saw things happening in computer vision and you had to translate it into weight spaces. And you know, and it worked, right? Like augmentations happen. How can we translate it? There's this perception loss. How can we translate into behavior loss? Right? And all those things. So you can borrow ideas from other fields. And it was an empty field to fill with content along those lines.

16:30

Hearing you talk about these identity transformations makes me think about like other kinds of geometric transformations in the weight space or like Cartesian, the polar transformations or things like that. We saw Google quantization, I forget the name of the quantization paper that just came out or actually it came out a year ago, but it was became, it was revisited a week or so ago and they did some Cartesian, the polar transformation, like all kinds of stuff that you can do in the weight space that you might. I'm curious how much of that is being explored.

17:18

So that's exactly so we came from this one area of motivation. We met the other people, as I mentioned, that are on all the symmetries on the group operations you can do. There's another world of. There's this mode connectivity that git rebasing where you can align models along some reference models. And if you just think about this permutation, you know, symmetries, there is so like a lost landscape that is connected. There's so many points that are the same, but they are different with respect to the weight space. And just understanding or trying to understand how, you know, lost surfaces looks like how models can evolve along orbits of this great work from BO and then Rose from UC San Diego on all this work kind of helped me to understand better what actually happens during learning. And then hopefully, you know, we could move this in our backbone learning backbone because, you know, some tokenization, some position encoding in the autoencoder is a transformer autoencoder. You know, we, we took some of the idea, it helped us. And then, you know, people are also discovering a groking and face transitions and, and how models suddenly kind of converge or suddenly don't work and then suddenly work. So this also is also connected. And I Would love to explore this direction more to kind of understand how can we make training much, much faster, better models or give guarantees is a strong word, but kind of, you know, bands of where models are operational with respect to their be like their performance accuracies, et cetera.

18:02

Yeah, I think the last time we covered kind of weight space, I don't think we talked about it as weight space learning, but kind of this idea of like introspecting weights was with Charles Martin, who had to work on weight watcher and crocking and mode collapse.

19:43

And so Charles. And Charles Cuauto is Michael Mahoney, who was then Qualdo in our paper. So yeah, yeah. And it was really funny because Michael was doing with Charles the work of analyzing weights and looking how they shapes are changing. So you can make a statement about if they converge or not. And they extended obviously really amazing work. So it also happened in parallel. I'm looking at this like an analytical piece of work where ours is like a learning piece of work. So we hope that we can at some point scale up our backbone and process more diverse model zoos, different types of architectures. We have a work where we are now able to train. And that's the vision. Right. Can you train on different sized models with different architectures, tasks, modalities, from open weight repositories or hugging face. Could we download everything from hugging face and train a foundation model of neural networks?

20:03

And you've started down that path. I think the.

21:11

That rabbit hole.

21:16

That rabbit hole. The poster that I originally saw at GGC that led me to you was something about kind of training on the hugging space model zoo.

21:18

Right, exactly. So after we were able to scale up with them, we're thinking like, can we kind of. Because we're still limited to model zoo. Right. So I can tell you I can generate now a new neural network, but I need this 1000 neural networks to have trained before I can generate that one. So you tell me that's great. But, you know, now we have 1001 neural network. So what are we gaining right at the end of the day, what are we winning? So then the next step was can we get. Can we train on models that are out there? And there's amazing work on analyzing how, you know, hugging face looks like the model atlas from Elahu, Horowitz and Yetit. And there is a lot of. There are a lot of models. So can we download those models and then independently of what kind of architecture they have or what data set they trained, use our machinery and it's A little bit tricky because different sequence lengths, different types of neural network layers. So we have to put some information into it. But we were able to train the first weight space learning model that can do generation and analysis of weights. So discriminative and generative downstream tasks on hugging face models. So this was work by Daniel Falk, which is really amazing. Again, we thought it's much more challenging to do this. But, you know, you have to scale. You need the machinery and you need those little tricks how to handle those different. You know, the tokenizer needs to be adapted to arbitrary architectures. That's the thing. Yeah.

21:27

And what's the filter that needs to be applied on the, you know, this hugging face's vast library of models that normalizes them to something you can deal

22:59

with, first of all. And this is work also that, you know, other figured out, like Eliahu. There's a lot of content on hugging face that's not documented. Around 30% of the models, they don't have any meaningful metadata. So you don't know what. Yeah, I mean, hugging face, the first

23:09

filter, just get rid of all of those.

23:27

Yeah. Knowing which models are helpful. So we need, we did a little bit of experiments. If we scale scaling alone doesn't help. You need to increase the diversity of the model zoom. So we want to have diverse models. So we want to have different data sets. Mostly focus on computer vision. Language is the next. And we kind of developed a scoring function on how popular is the model? Is it apparent or is it some derived work? There's a lot of trees in there. To download a set of two, we have 20,000 models, and from them, 2,000 models that are passing some quality checks. And from these models, the billions of parameters, you know, we trained the backbone on open weight models that then can sample all the different architecture. We can sample Vits, resnets, this entire thing was focused on computer vision. We were able actually to sample a GTP2 model. It's still a small model, but there is a domain change. And this model that we sampled we used as an initialization so it can train faster as compared to training it on regular data sets that are language. So there is some knowledge transfer happening from computer vision models to language models. That's also interesting because now we're still trying to figure out like what are those ways encapsulating. Right. And encoding. So that's kind of, you know, the next step would be then to scale it up and to train on different modalities, tasks and architectures.

23:29

You mentioned that there were some tricks that you had to employ to be able to use different types of models. And you mentioned specifically tokenizer. Dig into that a little bit more and also talk about some of the other tricks that you had to employ to do this.

25:05

So good models are important. Diversity is important. In the model weights you use for training. The tokenization and the processing of tokens changed strongly inspired by work from Kawang and in Singapore. So that helped us to identify or to process weights in an agnostic way so that we are not bounded by, you know, this is a layer starting, this is a layer ending.

25:24

And so to be clear, are we talking about like your, your thing is a model, right? And so it has its own tokenizer or are we talking about like normalizing the tokenizer of the models that you're ingesting or both?

25:52

So okay, that's a good question. We take the models that we download from hugging face, we strip away the weights in reading order very stupidly. There are probably better ways of doing it. So we kind of destroy the metric structure and then we just sequentialize or vectorize that and then it's a long sequence of millions of parameters, right?

26:09

Billions of parameters. Got it. So it's just numbers and then you've got a tokenizer as part of your ingestion process.

26:28

We also normalize. So normalization plays also important role. There are different positions where you can normalize on the weights as a pre processing during the tokenization or at the loss function. We tried different things and you know, currently it depends a little bit what kind of on the hugging face data we normalize instead of the batches during tokenization and other setups. I don't know. But this is important, the tokenization is important. We still playing around with the losses because we want to get some of the high fidelity high frequency information. So we're still kind of suffering a little bit with that. So just as a simple example, when we generate a model, the model is a little bit damaged. So we need some fine tuning steps to recover that. Very often these fine tuning steps are very quickly able to recover. But we given that our decoder has this problem of the blurry weights where we're kind of not getting perfect weights, right? So that's something that we are working towards. Because if you would think about. So let's think about the world where we could train on all this data, a foundation model of neural networks and sample on demand, your favorite model, whatever you need, then we would be actually replacing and totally replacing pre training, right? So why I have pre trained model you just sample the model that you need. So that's the vision of this foundational model of neural networks. We have one additional little, you know, thing that, that kind of causes also a little bit of trouble. We need to sample a model and anchor, where we sample and this anchor needs to be a model that we put through the encoder. So the better this model, the better the, you know, sampled models, which is leading to the situation that we need a well trained model to generate another well trained model, which doesn't make sense if it's in the same domain because, well, I have a model, why should I generate one?

26:35

Chicken and an egg problem. Right?

28:36

Exactly. So therefore we had this paper that we're going to publish soon in CVPR about remote sensing where we take an imagenet vit, use our machinery to generate remote sensing models or remote sensing foundation models. Then we have the domain change where our encoder decoder is providing knowledge transfer to generate models that are better than the ones that imagenet fine tuning would be able to reach. So here we have a true knowledge transfer which is really, really great, where we are able actually to outperform or be, you know, equally in performance with current models like telfm published at ICLR. I think the authors claimed they trained for 12,000 GPU hours and we are able to do this on 350 GPU hours. That's a fact of, I don't know, 20, 25, 30, depending on how you count. And this is suddenly interesting because you train from models and not from data. So if you think about we're running out of data, that's the reason why the scaling laws are a little bit, you know, considered differently. And everybody is moving into test time adaptation, test time training. We're running out of data to train the large models, but we are not using the weights of older models. So why not using the weights. All the knowledge that, all the compute that people invested, Right.

28:38

It's also an interesting context. Like if the, if all the data, if, you know, if we're in fact running out of data, at least in, you know, specific domains, visual text, et cetera, then all that data is already in a bunch of models. Why replicate that? And why not, you know, just find ways to, different ways to slurp it out of the existing models. It's essentially the, the premise, right?

29:59

Totally makes sense. In the remote sensing community we have 70 foundation models according to some surveys, and, and there are still people training the 71st, 72nd one. Right. So why not taking this knowledge, compress it all in, in a weight space learning representation and then sample models on demand. Because if you have a foundation model that is a VIT with 800, 900 parameters and somebody fine tunes it to a task, this person, the partitioner needs to use all the 900 parameters and that's demanding compute. And maybe the performance is a little bit better than a Resnet with 50 million, 40 million. So you know, you suddenly bound it in this foundation model world to large models. While with our machinery, with our way space learning approach, you could sample a big model, you could sample a resnet, you could sample an efficiency net, depending what you need. But you can give the architecture as a desired output and then we sample the parameters for whatever architecture which is helping for edge devices, helping for foundation model or helping for other use cases.

30:23

So you can also think of it as kind of a compression technique in a sense.

31:25

Exactly. The interesting question is what are we compressing, right? And how much redundancy is there? And can we do it in like pruning and also distillation train with that? Until now we train with the raw data, but we can generate smaller models. And these smaller models are better than if you would take the original model and do distillation with a teacher's student. So that's also what we show in this remote sensing scenario. So the goal is really like having one big foundation model of neural networks and then able to generate on demand models. There's one missing piece though, and we have a paper currently in review that might solve that. So if you think about that, we need a model as a prompt to get an anchor to sample other models can be a domain change. So what would be really, really nice if you would not need this model as a prompt, but you could prompt with your data set.

31:31

So give me a model that works well on this data.

32:28

Exactly. And if this could be done in a privacy preserving way, it would be even better, right? Because then suddenly people could use this without revealing individual members of this data set. Open reopen. So this work that does something similar already and kaist, you know, so Andreas and others who are working on that, and we kind of moved this forward because we have model zoos, right? We have data sets and we have models, so we have kind of images and we have models. And why not train an aligned space like Clip did with text and images? Then we could use actually a kind of a data set encoder as a model prompt that could move into this space and point into this space. Instead of a model prompt, we use a data set prompt so you know, you're a bank, you're a financial institution, healthcare provider, whatever. You don't reveal your data. You have your data set of, I don't know, hundred samples, thousand samples. You create one data set embedding. So you cannot infer, you know, the individual members or samples. You give this embedding to us, we provide you the weights, we give you the weights. You're much faster in continuing training. So this would be really interesting, right, because that opens up to all the, you know, potential data sets that are not on hugging face and is the

32:31

privacy preserving angle there? Because your process would start with that embeddings anyway. So it doesn't matter who prod produces it, or is that a compromise that you could do it with the processes, but you could probably get more out of it if you had the actual data.

33:53

Good question. So it comes by the method that we used because imagine you have a Data set with 10,000 1 million images. You need some kind. You cannot prompt with all individual images. You need some aggregate.

34:13

So you need to prompt with a

34:26

thing, one thing or one vector, right? And you need to aggregate this knowledge. So like, you know, we do a sentence and you have a text embedding to prompt your image that you generate. So it comes with that. Obviously you have to check for, you know, class membership attacks, you have to add some noise, et cetera. But the idea is really like, once we have this one embedding, can we generate from this one data set embedding now the tokens or the embeddings that generate the tokens for the weights? And this would open up, and then the question is, this would open up to data sets that people are not willing to share or not allowed to share. A financial institution that would maybe love to share, but they're not allowed. I have a, you know, external PhD student with the Deutsche Bundesbank, the national bank of Germany. They cannot share, but you know, they might use those embeddings for that reason. And it would also allow us to kind of understand how is the space that we're learning this latent space? Because the, the most interesting thing would be that we could interpolate the models that we see on hugging face, right? So can a data set that per definition is never on hugging face and there's no model that's trained on that. Can this data set and the data set prompt generate some meaningful neural networks that live between the space of known neural networks that are embedded? So do we have a well behaved latent space? Because suddenly if we could show that we have one. This machinery could be used for generating neural networks with minimum pre training or training at all. So think about the world where you could have on demand neural networks on demand hyper personalization in a forward pass.

34:27

I mean it sounds a lot, a lot simpler than the way we think about like neural architecture search today, which is a lot of very complex machinery.

36:12

Exactly. And to the end, I mean the external, you have to give the external signal. You want to wrestle it, what kind of architecture?

36:22

Right.

36:28

And we just generate the weight. So we are, I consider this work complementary to the neural architecture search.

36:29

Ah, so that might define the structure and you might provide the weights that fit into that structure.

36:36

Exactly. Yeah. Or you know, I mean, you know, I want to have this type of architecture, right. Give me a transformer, give me whatever. Right. Because this is, you know, per definition, per design already fixed and we provide the best weights.

36:41

To what degree does it produce weights? Kind of with knowledge of the, the architecture in which those weights will be used. Or is it just like you've got a parameter, you know, of, you know, the, the number of weights and it spits out the number of weights and then you have to as a post process, apply that to an architecture in a given way.

36:55

Exactly. So until now we with, with the model prompt you give an architecture. With the data set prompt you don't. So you need to give an external signal like give me a Resnet 18 or a VIP. But with the model prompt you give an architecture that we fill with weights. There are definitely more clever ways of building a tokenizer that kind of understands what type of architecture elements are processed. So this would be the next step to be, to be more knowledgeable about that and to also allow the decoder to generate conditioned architectures. We don't do this until now. We generate the weights. We hope that during learning and training the backbone saw enough instances of the same architecture. So we probably only can reco. No, we definitely can only reconstruct architectures that we saw in our hugging phase collection. But it would be probably much, much better if we could take this conditional signal on top. And some people are doing that, like people at, you know, in Singapore, Kawang and also kaist, they do diffusion on top of that. And then you have a conditional data set signal and conditional architecture signal. So this would be also the next step to go. But also under the, you know, promise, we can do it on open weights model that are diverse all the chaos. Right. Because that's where the most use can be generated from this idea of weight space learning.

37:19

Does the model produce a sequence of weights that you then have to map to a position in an architecture? Or does the model also produce like, does it produce a weight and a position in an architecture?

38:47

So it generates a sequence of weights that you have to then fit to the architecture. And every token that is then translated into a weight has a position encoding which position within the layer, which layer or which block, and then absolute position.

39:03

But that's on the input side, on

39:19

the input and on the output. It was an autoencoder.

39:21

Yeah. Okay, got it, got it, got it.

39:25

So then you get a sequence of weights and you have to fit it to the right architecture. You could fit it to a different architecture and then cut it or slice and dice it. That would be an interesting experiment. What happens then? You know, how many fine tuning steps do you need to repair that? And that's probably also one of the reason why we still need a couple of fine tuning steps to get the model very quickly up to performance. And therefore I think that conditioning would kind of help us to generate better weights quicker that are diverse enough. And so this would be potential next steps, right? To move and condition decoder and look into the decoder.

39:27

When you produce these weights, are you overriding an initialized model like random initialization or zero initialization or something so that the model is always valid, or do you have to think about, well, the model didn't actually hallucinate it and it generated two weights for this position and no weights for that position.

40:10

That kind of thing we entirely overwrite. So we replace whatever is in the model randomly initialized to have that, and we entirely override and load the checkpoint into that architecture. Since we have the sequence and we know the position coding, this is technically straightforward. Whatever the model does, it could have hiccups, have repetition. There's no visible pattern that would be like an artifact that we could observe that is kind of repeating again and again and again. But you could also think about not only generating one model, but because it's a forward pass, it's very cheap. You can generate ensembles of models, right, and then fit them and you suddenly have this new degree of freedom that you can generate. But yeah, we believe our ways that we can entirely override whatever is there. There's some interesting work that currently is

40:33

happening on task vectors and what is a task vector?

41:24

So if you have a base model, for example, and you fine tune to one task, another task, and the third task, you could take the difference of the fine tuned model to the base model and this would be a task vector and you could do task arithmetic with that. And in weight space people are using that in particular language models sometimes in computer vision. And when you work on those task vectors, we observe in some cases that it's easier to learn and easier to generate than in other cases. In other cases full weights are easier. So empirically we observed it, but we cannot explain that why for some setups, the one is easier or the other.

41:28

You mentioned a couple of times the distinction between high frequency noise, low frequency noise and the weights, which calls to mind like doing things in a frequency domain, applying an FFT to weights or something like that. Is that something that people are working on?

42:07

To the best of my knowledge, not. But I think what would be interesting is because what we generate is kind of this blurry base and then you could add some high frequency information on top. And I think in the image domain this is happening through you have a kind of a variation autoencoder reconstructing and then you have a generative adversarial network enforcing high frequency information. And then you combine lots of both of that. So you could do things like that in our domain on weights. I would not know if somebody really did this, but moving the weights into some other domain and then processing is an interesting idea. Not sure if frequency would be the right way of doing it, but you could think also a lot about a lot of pre processing steps until you would go and you know, do the weight space learning, compression or learning the latent representation, the lower dimension manifold on that. There are people who are also moving into that directions. But yeah, let's see, we're not going along that path. Yeah, currently.

42:25

And will the workshop be continuing this year?

43:38

There's another great workshop that is organized by some of our colleagues at icml. It's a workshop on weight symmetries. And we are thinking and continuing it for the next opportunity, which would be then neurifs. But let's see how this looks like. Currently we have a couple of people that were PhDs are finishing, new PhDs are coming. So there's a little bit of a gap in the community with the inflow. A lot of great people are currently finished with the PSGs on the market already in postdoc position. So there will be definitely a continuation on that. And like one of the most important experiences was recognizing that there is a community and once the community is there, you know, trying to develop a common language, trying to develop common benchmarking, trying to develop some, you know, ideas that are coherent as a community and moving along that way. I think this is now the time to try to move in that direction and bring those ideas together with the liberty of also going to the one or the other idea and direction. There's interesting work from Hage Maroon, for example. He's taking the ideas of weight space learning, applying it to gradients or applying it to activation spaces, doing representation engineering, which you could also think about the neural artifact that is not the weights, but, you know, the gradients during training or the activations. There's interesting work from Yedit coming around, probing neural networks. So not looking at the weights, but having controlled input output relationships and therefore looking, you know, what happens given an unknown neural network when they control the one thing.

43:42

Anthropic's been doing a lot of work along those lines as well.

45:31

Exactly. So there's quite interesting work that could be considered complementary to that. And it's always this one thing. Do we have these artifacts as a collection? Can we learn some representation and then use it to predict properties of the neural network or to manipulate and edit the behavior change and modify?

45:34

Very interesting work. Thank you so much for jumping on and sharing a bit about it with our audience.

45:55

Thanks for having me. And yeah, let's see what. Where we will be in three years.

46:01

Absolutely. Thank you.

46:09