Why Image Generation Needs More Than Bigger Models with Fatih Porikli - #773

Thanks so much to our friends at Qualcomm for their continued support and sponsorship of today's episode. Qualcomm AI research is dedicated to advancing AI to make its core capabilities perception, reasoning and action ubiquitous across devices. Their work makes it possible for billions of users around the world to have AI enhanced experiences on devices powered by Qualcomm Technologies. To learn more about what Qualcomm is up to on the research front, visit twimlai.com qualcomm Once a frontier research problem

0:00

in computer vision, text to image generation has reached a point where almost anyone can ask an AI system for a picture and get something that looks remarkably good. But looking good and being correct are not the same thing. Ask for several different people and the model may generate variations of the same face. Ask for a specific composition, identity or number of subjects and it may ignore those details. Push towards higher resolution or local generation and and quality, speed and memory quickly become constraints. The next frontier in image generation is closing this gap between plausible images and precise results. It's making these systems controllable, efficient and reliable enough to consistently produce a high quality rendition of the image you actually asked for one researcher. At the forefront of this work is Fatih Harikli, Vice president of Technology at Qualcomm, whose team presented more than 20 papers at this year's CVPR, the computer vision and Pattern Recognition Conference. Here's Fatih explaining why image generation still has plenty of hard problems to solve.

0:33

Maybe we are also asking a single model to solve too many difficult problems at once. Think about what happens when you generate a scene with several people. Now the model has to understand the problem, but I'm asking the model and then decide how many people should appear, determine where they should be placed, like the composition of the scene, reason about their interactions. Because if there's a person, if there's another person, most likely there, there is some connection. Preserve the identity of the person. We can give okay, this is my daughter, this is my son, and I want them to be in the picture, not like any random person. And finally, render everything together in all a single process. So maybe this is too much.

1:28

I'm Sam Sherrington and this is the Twiml AI podcast. For over a decade I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

2:15

Yeah, that's a big difference that I see between I think even this conversation and the conversation we had last year, the performance and capability of text to image models has improved significantly. And not just performance and capability, but I think accessibility. Like now ask whatever your favorite LLM is to generate an image and it will do a really, really good job. And it does kind of beg this question of the computer vision community, like, what's left to do if this problem is solved to this degree? What's left for us to work on?

2:37

That's a fair question. T2I models, these text to image generation models or image to image generation models have become incredibly good at producing very realistic images. The lighting looks natural, right, the details look right and the overall quality can be amazing. As many things we do, there's initial excitement, there's great work coming up. But still, if you look into that one dive deeper, you realize there are many things to still to be accomplished. One was, you know, like controllability, generating multiple people in the same image. People look almost identical, faces kind of blend together. And also we showed in the past we can run such models on users devices. You do not need to rely on a cloud service provider. I think most of the models still, still are limited to 1k 1k resolution. But then you want to go beyond that. That is a challenge which has not been actually addressed before.

3:19

Okay, so what I'm hearing in there is that we've made a lot of progress, but there's still work to be done. And when you think about that work, some of the big buckets include controllability, the ability to really get the models to, you know, focus on the way you describe the task or focus on the output that you want. And then you mentioned in there quality, so fewer artifacts, sharper images. And then you mentioned efficiency. We need to keep up with our ability to run the latest and greatest models on the device. So those sound like three chunky buckets for researchers to continue to work in.

4:21

Yeah, that's a very good depiction, Sam, thank you. You know, kind of, you said it very well. It's not only Qualcomm, but the community also trying to make sure the models, such models, generative AI models, can also match the guidance or the quality or controllability expectations or efficiency expectation goals of, you know, kind of like a real, like a camera. But you are taking pictures using your words, you know, your prompts. So there are still many things to be done. And that's why, you know, we publish, present many papers addressing such challenges at CVPR this year.

5:05

The first one we're going to talk to actually talks about this example you gave the diversity of Facial attributes. What's this paper called?

5:49

The name of the paper is DISCO Resolving the identity crisis in text to image D2I generation.

5:57

I'll mention to folks listening that a couple things, one, we'll have links to all these papers in the show notes but also we will. I'll try to get some of the the illustrations from the papers into the video. So if you're not watching on YouTube or watching video, look for that. Because all of these papers have really good illustrative examples that will help with following the conversation. And so Disco and Artican, when you think about them relative to these buckets that we've talked about, which of these buckets are they really going after?

6:05

They are about controllability and how to train a T2I model better aligned with the guidance from the user. Let me go a little bit deeper about disco if it's okay, Sam. When we look at, you know, kind of T2I models and our own models also what was interesting that this is not an image quality problem. The problem that I mentioned before, like we are asking the model to generate faces and certain number of faces and it keeps generating same faces, almost identical faces over and over again. So quality wise, image quality wise. If you look at the pixels and noise and everything, it looks realistic. But the missing piece was that those models, base models, amazing models had not really learned to create truly distinct identities because existing training objectives focus heavily on the realism and matching the user prompt. But they don't explicitly encourage diversity between people. And that is very important. So that observation led us to a simple question. What if the identity or facial appearance or any diversity, you know, kind of itself becomes an optimization objective? So that led into the idea behind Disco. Instead of creating a completely new T2i model, we kept the underlying model and fine tune it with reinforcement learning. We designed rewards that encourage several things simultaneously. For instance, different people within an image should have distinct identities. You don't want to, you know, kind of duplicate. Create duplicate faces. That's something we call in the paper as intra image diversity and across different runs of the same model with similar prompts, same prompt we shouldn't generate, we should not keep producing, generating the same faces. So this is inter run inter image diversity. So these are explicit new objectives. When we train, fine tune the model. And also, you know, we want model to generate the correct number of people. If I'm asking generate two people, that should be two, not three. We are also, you know, kind of incorporating that thing in the objective. And of course we still have the previous you know, image quality objective there. So putting all together, to put everything all together, we use reinforcement learning something called as group relative policy optimization. Grqo.

6:45

Taking a step back, it strikes me that you know, yes, this as an objective diversity of faces is an important one if you're generating images with multiple faces. But it strikes me as, you know, one of many, you know, possible

9:40

ways

10:02

that you might want to or characteristics of an image that you might want to influence. And it seems like tuning the objective for all of the possible ways that you might want images to kind of generate correctly seems, you know, not just difficult, but like anti bidder lesson. Like is it, you know, why not just collect more data with lots of faces and train the model with better

10:03

data in a way that. That is what we are doing. But we are. We show that you do not need a lot of data. And you are right, you know, kind of there might be many attributes diversity facial, you know, kind of diversity is one objective, number of people is another one. But then let's say we want to generate a certain action of a person or pose of the face or location of the person in the image or kind of their body pose and everything. So for those. But we are saying that, well, you can incorporate such objectives in addition to overall, let's say perceptual or know image quality objective when we are training such models. But you do not need a lot of data. If you the novelty of the paper point in this work, how to incorporate incorporate everything into a reinforcement learning framework that would make it possible and training fine tuning such models would be also kind of very affordable. By the way, there is a difference between the quality of the input images going into such models. There is another thing called as curriculum learning. Starting from simpler scenes and gradually increasing complexity makes learning this reinforcement learning much more stable.

10:33

I remember conversations I've had years, many years ago about curriculum learning. And it was always, you know, theoretical improvement. It's exciting to see it being incorporated into practical algorithms training algorithms nowadays.

12:01

Yes, you are absolutely right. Curriculum now is making impact. Not only by the way all our papers, but there were several other papers I see we are talking about how wonderfully curriculum learning makes, let's say multimodal models better. One broader lesson from the disco paper is sometimes the model simply needs the right objective in training, like the things that I mentioned. And also it is not mainly an architecture limitation, but it is how you are providing these objective and training data to the algorithm.

12:19

The way I would summarize your answer is that yes, there are lots of different attributes that you might want to exert some control over.

12:59

And

13:10

you know, there is a kind of a, you know, a mental or human cost to going after each of these attributes. But from a perspective of the training process itself, it's fairly efficient and nothing like when you think of like traditional fine tuning. It's, you know, very data efficient and you can apply techniques like curriculum to make it compute efficient as well.

13:12

You know, Sam, which something I like to also add because it is related to what we are talking about now. Optimization, of course is right. Objective optimization at the right data is definitely very critical. That is what thisco paper is talking about. But we may also kind of think that maybe, I mean, these are models, genetic AI models. Maybe we are also asking a single model to solve too many difficult problems at once.

13:40

That's kind of asking the question, could we possibly optimize for all the attributes that we care about? Are we in doing so, would we be asking the models to do too many things at once?

14:10

That's a good point. Maybe we shouldn't kind of. And also I can give you an example. Think about what happens when you generate a scene with several people going back to that core example. Now the model has to understand the prompt, but I'm asking the model and then decide how many people should appear, determine where they should be placed, like the composition of the scene, reason about their interactions. Because if there's a person, if there's another person, most likely there, there is some connection preserve. And you also, I didn't mention about this, the identity of the person we can give, okay, this is my daughter, this is my son, and I want them to be in the picture, not like any random person. And finally render everything together in all a single process. So maybe this is too much. And that's what we explored in the other paper, R2 camp paper. Instead of like asking why one model to do everything, what if we separated for instance, planning from rendering similar to what human artists might do. And you know, the kind of that kind of led us into this paper

14:21

before we dive into Artican. I thinking about that comment applied to disco. And again, this idea about different attributes, possibly a thing to keep in mind is how we've matured the way we think about like using model routers.

15:33

So,

15:56

you know, maybe you have, you know, a suite of models for the different attributes that you care about. And then when your prompt comes in, you've got some kind of router that says, oh, this, you know, the image that's required here, you know, will probably Have a lot of people let me route it to this model that's been tuned for that as opposed to trying to again, you know, have a single model that's optimized for all the attributes you want. It's a different approach to what you've taken with Arcan and we'll dig into that. But it is a way that, you know that DISCO could be put into production practically.

15:58

Okay, you brought up again another amazing perspective and we are working on it. This sounds like more like an agentic orchestrated image generation framework pipeline, right? Kind of depending on the input prompt. Maybe we want to generate for instance a realistic scene or like cartoony scene or something. Maybe there is a text or, or some like human generated graph or something like that in the image. So for all of it, different attributes like diversity, like facial identity, we may have specialized models, specialized processes within those models. So how to pull the right one is. We are working on such agentic also pipelines. Depending on the input, it goes, takes or determines the right tool. This is like if you consider this curve, one instance of, let's say diversity is one instance. The other one is better text generation. So it can go find the right one depending on, you know, where you want to apply them and then orchestrate the final generation. I think this is going to be the ultimate solution. You know, one, maybe size doesn't fit everyone. If you really, really want to generate something amazing top of the line. So we need such specialization. You are. Yeah, that's a very good perspective, Sam.

16:37

So I will again, the images with regard to this go are really interesting. Like I never really thought of, I don't know that maybe I've just never asked for an image with multiple people. But there are some really good images where you have a prompt that or several prompts where you're asking a variety of different models, GPT, nano, banana flocks, high dream, a bunch of them to generate multiple people. And you're absolutely right. You look at these images and it's like the same person cookie cuttered across the entire group. It's. It's kind of surprising that the, the models do that. When you talk about or when you think about evals. Beyond this idea of like running across the different models, did anything in particular jump out at you in terms of, you know, evaluation for this model?

18:16

Oh yeah. For instance, for this we look into existing benchmarks and then we pull together a new benchmark to specifically help people facilitate further research here and also, you know, kind of provide some standardization. We call it as diverse humans. It's a part, it's also kind of available in the, in the paper, the link for that data set, that benchmark. And you know, we invite everyone to take a look at it. So we look at that, we created this benchmark and we evaluated this on that benchmark and also on the other benchmarks, data sets. We see that when you explicitly impose such objective diversity objective, the score, for instance unique face accuracy detection score significantly improves. VSCO is around 98, 99. But you know, kind of like the models that, the base models where we started, that doesn't have such explicit diversity objective, they are very low, There is maybe more than 10, 20 percentage gap. So kind of this code, the solution that we also talk about in the paper sets the new sorter for, for diversity in text image generation. In addition to like human preference score, which is very important, evaluating the quality of the output is also superior to the, you know, kind of like the base model.

19:12

And it's interesting how you built the, the reward function and I'm imagining that you would do a similar thing from an eval perspective. So one of the components of the reward is what you call intra image diversity and that is, you know, looking at one image. Are the faces the same or are they different? And essentially you take your image, you apply a face detector that kind of identifies where the faces are with bounding boxes and then you kind of extract those faces and then you embed them and then you can just do similarity across, pairwise similarity across the faces and kind of get the distance from face to one face to the next to determine if they're, you know, all the same faces or essentially to get a measure of the diversity of the faces.

20:54

Exactly. Yeah, we talk about that in the paper. Kind of even there is a nice flow diagram. I mean, I invite everyone to take a look at it. As you said, there is a face detector applied and this is off the shop, you know, kind of. And then we take these detected ARC architecture AR faces, we embed them into some, you know, kind of a space, canonical space to be able to compare them. Then we compute such pairwise similarities and then it gives us a score for that image. What would be the diversity score? Of course, that is one score and we can do the same thing. Let's say I generated one image and I, I have another image with the same prompt. I have another image with the same prompt, but the initializations are different set points. So kind of the prompt is same, we run it and then we end up with a different image.

21:44

So you were starting to talk about the next paper, Artican and essentially how this model kind of breaks down the problem of text to image generation. Talk us through that.

22:45

So what motivated about that one disco is great. We as a pivotal example, we of course talk about this identity diversity. But then in the other paper we are making the point that maybe it's too much for a model to try to do everything going back to, you know, agentic flow. Again, maybe it's easier to approach some of these genetic AI challenges like a human being. We do not like try to solve everything ourselves.

22:59

Well, not just ourselves, but even like, even in this domain, you know, art like you an artist approaching this problem wouldn't necessarily think about it pixel by pixel. They think about like if you know what's the subject, what's the background and

23:37

there's a planning, right?

23:55

Yeah, there's a planning aspect to it. Yeah, yeah.

23:57

In this paper we build on it build on that idea. Instead of asking model to do everything, we separate planning from rendering, similar to how a human artist would work. So we have two components there, architect, which is it doesn't generate pixels, but instead it creates the structure or composition for the scene. Then kind of it says where, for instance, if there is a person is going to be in this image, where it should appear and if there are multiple people, how they should be arranged, it should look natural and realistic. Then artists starts with its composition structure and generates the final photorealistic image. While of course from the one objective also preserve identities if we provide identity. So yeah, this is like planning an architect, then an artist type of framework.

23:59

The underlying technical approach is also using rl just like disco. It's also GRPO base approach.

24:59

You are right. We also have this reward function leveraging on different objectives that we want to optimize. In this score we had inter intra image and intra group like interimage and human perception score and count accuracy here in R2. Can we also have. Well, we have the composition can be put right face into right place in the composition type of objective, which is a part of the reward function for GRPO and also pose of the face. Because now we are composing. I mean we don't want one people to look this way, the other people look the other way. If let's say we are taking a group picture, we could, you know, kind of, we could say that so kind of those type of things are all combined into the reward function, goes into reinforcement learning and then we kind of like optimize that one. The model now learns how to pull all of it Optimize all of it at the same time.

25:08

So to make sure I understand this, you have a prompt. The one in the example I'm looking at is three best friends riding unicorns on Mars. You pass that to the, what you call the Architect. The architect says okay, I want three faces and they're going to be here, here and here in this image and kind of plans out this canvas. And then that essentially becomes kind of grounding for your reinforcement learning loop. And so the images are all actually generated, but they're kind of optimized to ground to this canvas representation. Is that the right way to think about it?

26:15

Right. So that model, that architect knows where to position such faces. And artists rendering the base model where we started, now fine tuned to actually follow the guidance based on the locations coming from the centroid of those face areas. But there are two things to keep in mind. There is a training phase offline. So this model, the architect and the artists are optimized to do better. But in inference time, you know, first we take the prompt and if you want for instance a spatial person identity to be in that image composition, we provide those images and architect generates the locations of the faces. And there's no learning there, it just builds on what it learned before. And then the artist fine tuned model renders it. So this GRPO is on the fine tuning offline phase.

27:02

And so that's an important distinction for sure. So the, so the output of the Architect is essentially three tuples, three XY points in the case of three best friends on Mars, three people.

28:14

Yeah, exactly.

28:31

Right. And then that becomes input to the artist and then the artist just does a like a single pass inference or. Well, I guess it's a diffusion model.

28:32

So it's iterative, it's going to iterate internally. But you are right, it is you know, like single pairs in, in the sense that. Yes. It's not going to go called something else, you know, it's going to do it.

28:47

Yeah, it's interesting.

28:57

Why is that interesting? I guess it's interesting that it works because the artist only gets like locations. It doesn't get in the, in the training phase. The artist gets actual faces, right. Not just centroids. Or does it only get centroids and training?

28:58

Also if we want to say that there's a special person, there are special tokens representing a kind of like special person. If it is given, then we need to provide in training face those target faces. But you know, such faces can be generated and that's what we did also by another model. You Know, these are not necessarily real faces. I mean, to automate the training process. The training is very efficient. So in training time, those faces are given, the prompt is given and then target is generated. And then we evaluate like whether that is the correct number of whether the identity of the generated faces matches with the initial identity in addition to other objectives like count, accuracy and pose. Because pause is a. Could be a part of those prompt and also quality. So in training we are evaluating all of them, pulling them into this reward function. And then reward function through GRPO is optimized. And once this is optimized, now we artist knows how to generate better render better images given these inputs. And Architect already run and provided those, you know, locations.

29:17

And for this one, in evaluation, you have a bunch of prompts and then a bunch of faces that go along with those prompts. And you're evaluating several things in the images. One, are the identities of the provided faces retained in the output image. And also is whatever the descriptive scene or the prompt action reflected in the output image.

30:42

Right. We have to do all of it right. We want to make sure that we are still aligned with the input prompt. For instance, if you say three best friends riding unicorns on Mars, like three. The example in the paper still we see three people and there are unicorns, you know, kind of. And it's like the scene is like Mars. So that is alignment to the input prompt. We still impose that. We also want to make sure that the generated image is high quality. This is human perception score. And there are three people, not like two friends. So conflict is there. And also, okay, I mentioned three friends with specific friends, not like random people, like people riding unicorns with three, you know, kind of Sam, me, you know, another person kind of like. So we actually provide some examples of our faces. So we also want our pictures to be also our faces to be in the generated picture. So that's the identity face matching objective. And also another one, we want them to maybe look at the camera. If that's a group picture. I mean, you. This is again one example of how to improve the image generation. It doesn't mean that this is the only solution or, you know, kind of like Qualcomm is showcasing what is possible.

31:13

Is the identity retention like a big part of the kind of the value of this approach, or do you see it as a. As a valuable approach independent of identity retention?

32:38

Oh, absolutely. Again, the composition itself, it would look much more natural because we ask Architect to tell us about the structure of the image. Definitely. But, you know, kind of Like I said before, the message is bigger message is, hey community, everyone working on image generation, text image generation models, look what you can also accomplish if you really think smartly about the objective function like the disco paper and also simplify the task like R2 CAM paper. Not trying to do everything at once, like I'm going to generate the composition and the render, but divide those processes into more manageable parts.

32:51

So we've got a couple of papers as well focused on more of this kind of quality Bucket. Disco and Artican also are, you know, both controllability and quality. But the, the next couple we've got here are Pixel Rush and Inverfill that are really squarely focused on output quality. Talk a little bit about what these two are doing.

33:37

Absolutely. The first two papers, like you said, were about making image generation more controllable and like incorporating, you know, different objectives and making it easy for model to do the things that we expect it to do. The papers you mentioned, Pixel Rush and Invertfield, shift gears into a different objective. Now we ask how to generate images much more efficiently than even what we did before because Qualcomm proposed and presented many papers at the previous CVPRs how to run such models much more efficiently on mobile phones. But now you are saying that can we even push it to the next level? Because before we have been talking about just a 1k resolution and that's the current software. Even the cloud models are kind of limited to this resolution. But the question is now can we do like 4 megapixel image generation, 16 megapixel image generation? Of course the challenge is not only, you know, kind of like how fast you can run the model. By the way, such models, existing solutions may take anywhere from, you know, kind of seconds, like 50 seconds to minutes. They are not that fast. But there is a memory challenge also because when the image resolution gets larger, we need to retain this diffusion process, the latent features in the memory somewhere on the device. So that requires big footprints. So how can we run a model such that we generate a extremely large image, high resolution image and of course the quality has to be also still, you know, real high resolution, not just, you know, kind of like high resolution up sampled image, but real details, a lot of details are there. And it would run at a reasonable time. You don't need to wait like 10 minutes, which is the current, you know, kind of, I mean, XFR solution, the time such model states requires and it would run on, let's say a memory available on a handheld device like a Smartphone. Yeah, that was our objective for the let's say pixel Rush. And also we didn't want to change really the existing models significantly. So that is something to note. There are wonderful image generation models and you know, some of them kind of like by many big companies, kind of. We are all, we all know those models. We didn't want people to go and have to fine tune those models. But we are saying that hey, you can still use any of those models that you have and then follow our pipeline we discussed in the paper. So you can use that model, still generate let's say four times, 16 times more pixels.

33:59

And so talk a little bit about the generation process. Like what, what's different about the way you've approached this?

37:14

Yeah, absolutely. So it again starts with a prompt and there is this base generation. Like any model, it could be like say flux model and then it generates let's say a base image, but base image. What I mean by that, let's say 1K image. And then we have this cascade up sampling stage. That is the part that is null about this paper. That's why I say CPR paper. This cascade up sample what it takes, it takes this image which is like RGB pixels, not latent spaced. And then it creates for instance any using any apps image super resolution solution. It could be bicubic up sampling or know kind of it could be something smarter. It generates let's say higher resolution image. So when we do that, okay, we have now let's say 16 megapixel in the image, not one megapixel. We have like a lot of pixels and then we take that image Sam and then we apply an encoder, a vae. Then we go into a no kind of latent space. In that latent space. Of course I mentioned that we want to. We are concerned about the memory. We now divide that latent space into manageable chunks. We patchify them. So then we do improve those latent space features. But then we are still in the latent space. We go to the, through a VAE decoder in this case to the pixel space. Something we need to be very careful here. Yeah, there are solutions also, you know, kind of using pagefication like I'm going to take and create a you know, kind of like patch, then another patch, another patch. And when you do that you create artifacts actually, you know, like scenes visible.

37:21

Meaning when you do that in your origin space, you create patches. What's different here is that you're doing it in a latent space.

39:21

Absolutely. There are many reasons One is latent space is much smaller spatial dimensionality than the original pixel space. The other one is in latent space we can induce nodes. And that is a smart way of leveraging noise. Yeah, that is the other reason. Yeah.

39:29

And so you have in the image describing the cascade coarse latent refinement stage and then high quality latent. Are, are these two separate latent spaces or is it one latent space? Like what does this mean? Help me understand, like wrap my head around.

39:56

Latent space is the, you know, latent space of the. A kind of original map of it is not a separate space.

40:17

So what does the refinement stage do?

40:26

Refinement space that adds some guided semantic noise and then it iterates a couple of times. That is the new part, that's the part that we provide. So it starts with these latent features. It gets some additional kind of flexibility through this injected semantic noise. We want that because we don't want suddenly, you know, kind of, okay, we are generating a big image, but then there isn't enough texture in the image or high resolution, semantically meaningful texture in that image. So that's why we want model to have flexibility to add things. We are in the latent space and then we induce such noise. I mean, we add literally some guided noise into this latent space. Because diffusion models, they, the ones that we are using, they do diffusion in the latent space and they start with noise, like random noise and then iterates, iterates, iterates like many, many times. I'm talking about like high level idea of the diffusion. And then every time it estimates some noise and removes it from the original previous noise. And then it clarifies step by step and then it ends up with a final image which is now noise free. It looks like a real image. That's how diffusion, denoising, diffusion, text image models versus work. So kind of we are in the same space, in the latent space is not the pixel rgb, but these are specially lower. Basically input is kind of 1k, 1k. This is maybe 128, 128. And of course at every, let's say point we have a vector representing the, you know, features for that page corresponding to a large pixel. So that is the latent space.

40:29

Is the dimensionality of the latent space larger than what you might use for smaller images? Or is it the same?

42:37

Yeah, latent space dimension is same. That's why this method is still very efficient. I mean we could of course go to larger latent space state, then the

42:44

but that has a cost.

42:55

But the memory challenge is there. Right. And also it will take forever. The baseline actually does that, you know, kind of like they just make the latent space larger, proportional to the target output resolution, which is not. We wanted to do, we wanted to run everything much faster. Yeah, that's why latent space is much smaller. Like the original 1k 1k image latent space.

42:56

So conceptually, the way I am thinking about what you're doing is you generate a low resolution image, you apply an off the shelf up sampler to get a larger image. But that image is gonna be kind of blurrier and not all that great because that's the state of the art for up sampling. And then you essentially go into latent space and you, you like, you end up in the blurry place and then you apply diffusion to kind of generate out a higher definition.

43:21

When we go to latency space, there is a nuance and it's important. So we up sampled the image input and this is VR rgb. Like this is like color image. Right. We have now larger version of is large. If I go to the latent space of the same proportional size to the depth size is going to be very slow. So what we do, we partition the input image into like original size patches. Let's say I started like 1k 1k image, now I have like 4k 4k each side. Like so I divide it into 16 parts. So I have like 16 1k 1k images. And then I.

43:59

Do you project each patch into its. Into a light and space.

44:40

Absolutely. Okay. Yeah.

44:44

Got it.

44:47

The nuance. Yeah.

44:47

Okay. And then, so then how do you, how do you avoid artifacts at the borders when you project out like.

44:49

So these images are not really super resolved in the way that we wanted because they are just up sampled. Right. We really didn't add any semantically meaningful details. We still have the original prompt, by the way. And that's why we add noise for each latent space.

44:59

Yeah, that's why I think about it as like applying diffusion, you know, text driven diffusion to this, this, the latent.

45:17

Yes, yes. And so it allows us to create more texture. The algorithm is going to definitely generate much nicer looking output image than the input image. Given input image. Before we went to the latent space. So in the latent space now we provided this freedom for algorithm to generate better semantically meaningful texture. But Now I have 16 pages, right. And if I stick them together, arrange them in the original arrangement, that will be because they did it independent from each other. Right? Because of the memory. And of course we allow some overlap, small overlap, but it's not going to solve it. You know, kind of Bambi page Y by the way, another nuance is those pages are overlapping slightly, but it's not going to solve our problem. So what we also realized that when we are doing blending across such, you know, latent space data representation for these 16 pages, we like to again do another noise injection. Then we combine them together and. But in this case it is not like all over the patch same in the latent space, kind of like sample from the same distribution. We kind of allow algorithm to generate more noise towards the boundary, but in the center of the pages, you know, kind of like be maybe less, at a less lesser degree. Now we generate these high quality latent which also at the same time now resolve added texture but also resolve the boundary. This seam issue because we had so

45:26

you kind of like. I'm imagining that as opposed to like independent independently processing patch by patch, you're like pairwise or crosswise refining these patches. So that on the other side of that refinement process, they're kind of, they're aware of each other in the sense that like they, the edges are aligned and blended and all of that kind of stuff.

47:14

Yes, but when we generate this high quality latent, most of the problems already solved. The it will allow when we go to VAE decoder, generating this high resolution image which is, you know, now artifactory. I mean we solved the problem. Handle the test at a space in the latent space where it is easier to do. Because if you go to the image space, if we start blending into image space, we will most likely blur around sims, which is another artifact we don't want.

47:43

Well, it's amazing that that works.

48:20

Oh yeah. I mean again, like the all our motivation in this case, let's achieve a quality level. Like we have a big memory and a lot of compute power and we can do everything into a bigger latent space, but we are patchifying and do it maybe much faster. When I say much faster, it's not like two times faster, it's maybe 35 times faster, you know, from let's say 10 minutes to around 20 seconds type of, you know, acceleration.

48:23

And it's interesting that we're talking about kind of boundaries and images and like reconstructing them kind of correctly because that's part of what the inverfill is doing also, isn't it?

49:01

In inverfil we also have a semantically seared noise injection, like the pixel rash. But infil is a, is not T2I model. It is an image impainting model. So this is image to image Generative AI for instance, you can touch an object, create a mask and then remove that object. Or you can bring another object, new object into the scene and then you can create this. Very practical Sam, because many times we take, we take pictures, but in the background maybe there are things that we want to remove. The challenge is, well, I can still see, you know, kind of you create a texture sometimes, okay, it is meaningful.

49:13

But then there's those artifacts that you see in the background. Like the, I see the sand on the beach where you remove the person who didn't need to be in the picture is kind of funky looking and

50:00

maybe the texture is not really compliant with the rest of the image. Or I see literally the artifacts around the boundary. So kind of now, yeah, it is like a toy, you know. Yeah, you can do that, but I'm not going to really use it. But we are saying that, hey, you don't need to be there. You can do much better object removal or imaging painting, image editing. And that's what this invert fill paper is talking about.

50:12

And so what's the core idea behind this paper? I'm imagining that you're using diffusion, you know, maybe multi step or one step diffusion somewhere.

50:39

So in invert fill, what we do, okay, we have this input image and we allow generating noise in the background. And then we have also this noise in the mask, like the bird that we want to remove. But what I mean by like this noise in the background, the rest of the image.

50:53

So

51:17

in denoising, we started with, during training with a noisy image and then we progressively remove noise and end up with a clean image. In image generation we also do the same thing, right? We start with the noise and then end up in with a clean image. But think about the other process, like the way that we actually train those models. Those models, when we train, we start with the clean image and then add noise, noise, noise. And at the end it becomes like noisy image. So think about the reverse Pro process where we actually train the model so we can take this input image and map into noise. So we are progressively inverting a real, let's say clean image into noisy versions. So kind of this thing very well studied, understood. And it's very fast, like 60 milliseconds. We can take a large image and then end up create going through the reverse denoising and a noisy version of it. So this noise is not random noise anymore. It is specific to the input image. So if I change the input image, the noise is going to be different. So this is about entire image. And then we have this mask of the bird. Now I added noise to there for the bird because I allow. I want to allow algorithm to generate a new bird compliant with my field prompt text prompt. So now I changed the way that we kind of create this image. But I don't need to change the original model. I can still, you know, I changed the way that I initialize the diffusion in this painting. And when we do that, when we start with you know, like this inverted noise plus the, you know, kind of mask and new noise within the mask. Now first of all we can retain background, but we also allow background to slightly impact the foreground like the mask itself. So allows seamless harmonization and it generates, you know, high quality images. But most importantly, there's no boundary artifacts anymore because we have two noise, you know, like noise, like random noise and inverted noise within the same. When we started the image, we are not starting from like just noise within the mask.

51:20

Awesome.

54:02

Awesome.

54:02

So yeah, as we suggested starting up, you know, Qualcomm always has a ton of papers at cvpr. We can't cover all of them. We've covered a handful of the most important image generation papers. But there were also papers on video generation. There were a ton of demos. Anything in particular you'd want to call out in terms of other things folks should look for?

54:03

Absolutely. I'm very proud of the three video generation papers because it makes video generation accessible to everyone. You can run such models now on your PC laptop or your phone. You know, kind of those three papers are. I will just mention their name. The first one is Pre Middle wan. WAN is one of the best open source available, publicly available model. So we are showing in that paper, hey, now you can actually run it much faster and you can like five times faster and then put it into squeeze it into your memory limited device. You don't need to run it on the cloud. The other paper is hybrid attention rehired. So kind of recurrent hybrid attention paper. Again like there's this challenge of where to apply attention. So it kind of does much better job and makes it faster. And also attention surgery because attention takes a lot of compute. So these three paper for video generation, amazing papers. Please take a look at those project pages. Also we have project pages found for these papers.

54:31

Well, Fatih, as always, it's been great catching up with you and hearing about what you all are doing with rock. Regards to computer vision and all the cool things you did at cvpr.

55:48

Sam, thanks for having me. It's really a real and always, you know, a pleasure to be a part of your amazing podcast. And I'm very excited to be a part of it. Talk about artwork. I only mentioned a few of them and I look forward, kind of meet again, you know, join your amazing, amazing podcast. Yeah. Thank you so much for inviting me. Alrighty.

56:02

Thank you.

56:27