One thing I find really interesting about the DALL-E-2 is that the popular blog name ("DALL-E-2") never shows up in either of the research papers that describe it. The paper commonly referred to as DALL-E-2 calls its own algorithm "unCLIP". UnCLIP is _heavily_ based on a paper from a few months earlier called GLIDE - in fact you can't really understand the unCLIP paper by reading it without first reading the GLIDE paper.
I suspect what's going on is that OpenAI has decoupled their PR activities from their science activities. They told the researchers to publish papers when they're ready, and then the PR apparatus decides when one is good enough to be crowned "DALL-E-2" and writes a blog post about it.
This is only surprising if you're not familiar with product launches that result from R&D. In this case DALL-E 2 is the consumer-facing name, unCLIP is the name used during research and in this case for publication. OpenAI may also have a further internal codename that they used for the project. Currently DALL-E 2 access is limited but there are lots of reasons to believe that OpenAI will try to productize Dall-E 2 as an API. If you're selling a product, you need a product name.
I honestly hope they don’t productize it until they can build a dataset of images that have been explicitly licensed for remixes without attribution. Having proven that the system works, they could now justify the effort to build a very large public dataset of CC0 images. For example they could encourage Twitter to add image licensing options and use the alt text of images to form the image description. They could then encourage twitter to encourage the use of licensing and alt text, and foster a culture of open sharing. There are already 500 million images in Creative Commons, hopefully many with an appropriate license for reuse without attribution.
Otherwise what I am hearing is that artists online are upset their work has been incorporated in to DALL-E without their consent. A system that rips off artists and then puts them out of work would make OpenAI and the AI community look bad.
Instead we could move forward with better licensing and alt text features for major platforms and draw in an open community rather than taking what has not been licensed for this kind of use.
I disagree. If it’s on the web it should be fair game. These models are not directly cutting and pasting, it’s my opinion that they’re learning some higher level features and attributes. Admittedly, not everyone shares that opinion. Humans don’t exclusively learn from licensed content and I don’t see why our learning algorithms should be restricted to that.
Why as an artist should I assist in my own demise? If you are coming gunning for my way of life and skill I've spent half a lifetime developing, my very purpose, don't expect me to just roll over and help you.
We still own the rights to all our material. I don't see why we can't en mass expressly forbid the use of any of our creative works in machine learning training sets. That's a specific and measurable point where our material is inserted in to a machine without prior permission and with malicious intent towards us.
It seems to me there is a gap in robot rights vs human rights:
Google has scanned in much of the world's library collections - the search agent is allowed to look at all of it, but I am only allowed to see samples. How is it Google is allowed to keep a copy on hand, but is not allowed to show me?
And now OpenAI has done the work of curating hundreds of gigabytes of publically viewable images (copied without regard for copyright, since of course that's how the internet works, but they have made a copy of each of these images for their training set) -- it seems strange to me that if I want to ask DALL-E, "what image in your training set is the nearest match to this output", copyright law prevents it from showing me (perhaps it could at least return the URL it recorded, and I can hope that Archive.org has taken on the risk of violating copyright on my behalf)
In any case, intellectual property laws have created a world where the robots are compelled to withhold information from us mere mortals, with the excuse that they can't make one more copy for us.
Is it really that surprising, though? The training data is more like 'inspriation' for human artists. Certainly, there are cases in music where sampling or melodies are a little too close to the original and spark lawsuits, but largely, very similar derivative works abound.
Nothing about creating a derivative work, intentional or otherwise, would allow you to use the original in a way copyright protects. Even Fair Use, which is fairly permissive, doesn't allow for blatant disregard of the original copyright.
That's an interesting thought experiment. Every human btain contains illegal copies of music and images. If you hum or sing or even think about an IP, in the future you will have to pay a license fee to a corporation.
It’s already illegal for you to sing a copyrighted song in front of a group of people. If you hear a Taylor Swift song and memorize it, then later sing it in front of a sufficiently large group of people I believe that is illegal.
Now I don’t actually think intellectual property restrictions are good. I think they are quite harmful. But as long as they exist I don’t like the idea of big companies trampling over the IP rights of small creators. People currently depends on those rights. One of the reasons I want to see a large public dataset of explicitly licensed images is that if OpenAI is required to do this, they would have to spearhead a new culture of openness to get enough data. Imagine if they encouraged twitter to add a license option and then simultaneously encouraged users to add alt text and to license images with an open license! This would be a boon to people with sight disability who require alt text and it would expand a visual commons of openly licensed data. Instead we have one company making a big dataset full of copyrighted images (sharing something online does not waive copyright). I think a model based on explicit consent is healthier for everyone in society.
Same with all these big models. GPT-3 and codex benefit from all the text and code that has been made publicly visible without giving back. At least e.g. imagenet classifiers had to take the effort to label the images they scraped.
IP laws need to catch up, so that ML models that use unlicensed public content have to give back in some way.
I noticed that as well! It confused me a bit at first. They say that their "image generation stack" is referred to as unCLIP, and I was trying to figure out how it's distinct from DALL-E 2 at first!
My only guess would be that unCLIP is the end-to-end image generation model, but if the model is used for manipulation, interpolation, or variations, then it is referred to as DALL-E 2. So unCLIP is a subset of DALL-E 2.
Maybe it's because there is more features on the openai websites?
for exemple with GPT, you get different models, different templates, a playground, an api etc...
HN's markup is idiosyncratic but similar enough to markdown that it's hard for occasional commenters to remember the details. It's also minimal enough that users are already used to parsing extra-syntactic markup visually.
Interesting. It doesn't work for me; I have to remember that underscores parse to italics and then "apply it in my head", rather than just reading italics, which gives them a distinctly different quality. For example, I just heard your response in my head as "Underscores still work", which is different from "Underscores still work." They also don't pop out as much when scanning text, as italics do visually. I have to spot the underscore before and after the word, and it really doesn't work if you try to emphasize more than one word. At least with the ">", it's at the beginning of the line and in a separate paragraph.
I think it's a combination of Markdown being completely readable in plain text especially if you're familiar with the syntax and even if you're not. Similarly, I see a lot of people using TeX-style mathematics, it's not particularly readable but "The quadratic formula is $x=\frac{-b\pm\sqrt{b^2-4ac}}{2a}$" is a decent way of representing the formula to people fluent in LaTeX even in plaintext conditions. I suppose there's also likely to be a bit of muscle memory where people accustomed to typing in Github/Stack Overflow/Reddit markdown use it on other systems, and even if they see it's not supported it's good enough to not need editing.
I don't think it's particularly worthwhile to learn a new comment format (one that's not even linked or described in the comment editor, for that matter) for every site.
I don't know why you're being voted down, internal/research names are often way weirder and decided on ad-hoc by the researchers themselves and then a good PM comes in when product-ionizing and part of this is deciding on a catchy name for public use.
The phrasing "I don't get it" is fairly rude - it implies the post is obvious to the point of not being worth mentioning. However obvious this might seem to somebody, I would point out that turning AI research papers turn into products is hardly commonplace.
This behavior is not exclusive to OpenAI. NVIDIA did this too. Originally StyleGAN3 was published under the name "Alias-free GAN" and the paper itself uses that terminology.
Gaming is going to get so interesting with these emerging technologies. I played A.I. dungeon sometime ago and I was amazed at how good it was at making up believable stories on the fly.
Now imagine joining this with dall-e and you truly have a game which has never existed until now, with it's own story and graphics that you are creating on the spot.
Unlike the adventure games like King's quest where everything was pre-programmed, this is truly infinite never-ending game with a unique experience for every single player.
Like the guy from the '2 min papers': what a time to be alive. I feel so happy and excited just thinking about the possibilities these techs are going to bring.
Yeah, it's getting to the point I'm starting to see current game design as getting long in the tooth by comparison to what I know is ahead.
There's a great tech demo a dev did a year or two ago showcasing GPT-3, speech-to-text, and text-to-speech to have random NPCs in a VR open world respond to anything the guy said if he walked up to them and talked to them.
Procedural generation has taken on almost a "dirty word" reputation in the past few years in gaming, but as AI continues to allow for exponential variety at increasingly high quality, it's going to enable some truly mind boggling experiences.
Expect to see MMO models (subscription fee and server-oriented) but for single-player instanced worlds dynamically generated around your interactions in them.
I can't wait to have a party of friends to go on epic adventures with that are all just AIs I picked up across a world along the way.
Less than 20 years away, and possibly even less than 10.
Can you elaborate on why it's a "dirty word"? I see a ton of potential with generative/procedural game elements, esp given all the recent AI breakthroughs. But I am more of a former gamer than current gamer and am surprised to hear it's a dirty word.
"Procedural generation" usually means content generated by a human-crafted procedure. Given that a human made the algorithm by hand, its scope is limited and easily becomes repetitive.
When I played with it, I started a quest as a wizard looking for a book.
I was able to cast a tracking spell that led me to a giant library.
I could have it read the titles of the books on a shelf in front of me.
I could pick up any book and open to a random page and have it tell me what was in it.
One was about a little half-elf that had a magic flute that broke.
I cast a summoning spell to summon the half-elf and fix its flute, after which it happily played a song opening a door to another dimension filled with musical instruments.
Give me that level of emergent gameplay in a VR open world, and then just take my money and all my free time, as I'm never leaving.
We're simply very, very early on in what's arguably going to be the most transformative tech since the Internet. People predicted back then that the slow network which only offered basic things like email wasn't going to significantly disrupt things like retail.
They were only right in that it didn't remain slow and ended up doing a lot more than email.
This stuff is getting better way faster than any tech I've seen, and I used to consult for CEOs at Fortune 500s and sit on advisory boards on the topic of emerging tech.
I wouldn't be so quick to bet against it. We really haven't even started to see what these models can do in application.
Yes! So many exciting possibilities in so many industries. Hopefully it won't displace artists though, we'll have to find a way to manage the efficiency of AI with the curation of artists!
On the other hand, it will empower and enrich people that are especially good at creating prompts that generate good content. Who knows if more jobs will be created than destroyed.
> I feel so happy and excited just thinking about the possibilities these techs are going to bring.
Like what?
I should preface by saying I think art is great, I know a lot of artists who struggle to make a living, and it is somewhat heartbreaking to think of all the poor art students who I guess we should pay for their educations and will never have careers now?
True, but artists also sell artwork - my question could be reframed as, if Dalle 2 can produce a Rembrandt, is a Rembrandt worth anything, even emotionally?
I think it's worth pointing out that DALL-E 2 mimics the style of famous artists. The artists had to come up with the original style in the first place!
There are highly competent artists that can create highly convincing copies (fabrications? forgeries?) of famous paintings. Are these paintings worth anything? No, because people find value in the specific contribution to the field of art that the particular painting represents.
I think we should look at DALL-E 2 like a highly competent artist that can produce convincing forgeries and even mimic the style of famous artists, but cannot replace the artists themselves.
> I think we should look at DALL-E 2 like a highly competent artist that can produce convincing forgeries and even mimic the style of famous artists, but cannot replace the artists themselves.
We don't know that. DALL-E must encode its knowledge about art styles somewhere in a way which permits applying them to almost arbitrary contents, similar to style transfer. So, what do you get when you 'just interpolate' between artists in the high-dimensional space defined by a multi-billion parameter model? It is vanishingly unlikely that that point in the style latent space will correspond to the output of any artist who has ever lived. What if you manipulate the encoding to push it far away from the latent points of a large set of real artists? And so on. We don't know what that looks like or if the new styles would not be considered 'original' by critics unaware of the machine origin.
I don't know that the metaphor works out in your favor here... if you were to catch an art theif you might send them to prison. If you have automated the process of producing art theives, that seems like a problem you have caused. You HAVE replaced the talented artists, because you didn't need to consult the original artists.
How does this benefit society? Even in the case the artist themselves uses the generator, is the artist better off for no longer needing to create art? It's not like there is an economic advantage, and so there isn't as much incentive to create even original art anymore, someone can just immediately mimic it.
People mimicked art in past, but at least for the artist there was the value of honing their own skill, and remixing or enhancing it, maybe tools like this have the potential to destroy creativity itself.
Only if you're sure it didn't memorize a copyrighted input, and only until everyone gets bored with its style or you want your assets to look the same in a predictable way.
We will still need people with taste to drive the machines and curate output.
Also, like, this is how the world works. To cite a hackneyed example, people who worked with horses had to figure something out when new tech displaced them. So will graphic designers, illustrators, et al, if indeed AI is a more competitive option for their services.
Not just graphic designers. In NLP, what used to take years of data labelling, architecture design and model training now is being done zero-shot by GPT-3.
Simple automations can be driven by GPT-3 as well. It needs a representation of the screen and it will automate the task described in natural language.
AI generated images are not art. They might use the same medium visual arts do, but they lack a meaningful vision of the world.
Of course defining art is a subject in itself, but I think that being afraid of AI replacing artists is comparable to thinking photography would when it was invented.
So we should halt progress just so people can keep their jobs? I hate this argument whenever AI or automation is brought up, it's probably one of the worst ways to deal with it.
As long as it is progress. What usually happens though is we get a watered down version of what we had before, but since it is cheaper and far more profitable the big companies exploit it to maximum effect. So in reality we lose a lot.
No thought is put into the effects of technology. You can have progress without trampling all over people and their livelihoods, but we don't often see any effort to do this.
Why are we focusing on image generation as the target for AI automation? It's not like there is a shortage of artists, expect perhaps amongst programmer bros. Why are there no efforts to automate away positions such as executive offices? Are CEOs so much difficult to replace than artisans?
"I invented a method to kill people, what are supposed to halt progress because more people die now? I hate it when people bring up that I'm a serial murderer".
There are things I would be doing with AI and things I would not. Being happy about making a even just a particular type of art redundant, well it if it doesn't benefit anyone then why would you care?
Besides, we have banned certain types of research before, precisely because we decided they weren't ethical. Your argument for "progress" is a poor one.
> No thought is put into the effects of technology. You can have progress without trampling all over people and their livelihoods, but we don't often see any effort to do this.
Sure it is, and sure we do. There are ethics in AI boards for a reason at the major AI research companies, as well as many other technology-producing companies.
> Why are we focusing on image generation as the target for AI automation?
Who is "we?" Individual researchers focus on what they want because they like that topic of research. There is no committee out there who figures out, "what industry can we put people out of jobs this week?" No one wakes up thinking, "oh boy, time to put artists out of work."
> Why are there no efforts to automate away positions such as executive offices?
There are. See abstract thinking AI research.
> Are CEOs so much difficult to replace than artisans?
Yes? Abstract reasoning, allocation of abstract resources like engineers (not widgets or manufacturing) is not an easy task to create an AI for, if a CEO is automated then that means we've reached artificial general intelligence. In contrast, like you see with DALLE and others in this link, art is much easier to automate. This doesn't even get to the notion of artists being in a surplus, they choose to do art as their livelihood. Now, you could ask whether it's fair and whether we need a UBI or something like that, but that's a different question entirely.
> "I invented a method to kill people, what are supposed to halt progress because more people die now? I hate it when people bring up that I'm a serial murderer".
Funny you say that because the military has invented many things that are now used in civilian life [0]. Even the smartphone you use is in part due to their innovations. This is not even to bring up nuclear weapons, which while horrific, have effectively been used as MAD, not to mention a highly efficient energy source via nuclear energy.
> There are things I would be doing with AI and things I would not. Being happy about making a even just a particular type of art redundant, well it if it doesn't benefit anyone then why would you care?
You don't speak for everyone. Why wouldn't automating image or video generation be beneficial to "anyone?" I could imagine at least several use cases off the top of my head, such as a unique streaming service for every individual. Artists themselves can also be influenced by such media, it's not like they stopped after photography was invented. Art just changed.
> Besides, we have banned certain types of research before, precisely because we decided they weren't ethical. Your argument for "progress" is a poor one.
Perhaps we shouldn't be banning research just because of some subjective morality. You've shown no argument for why "progress" is a poor incentive, merely that you personally don't know or care enough about it.
And one more thing, art isn't done as a livelihood, it's done because the artist has a deep appreciation for the works they want to create. That some do it as a money-making endeavor is immaterial to this fact.
I'll just never understand how any of this works. I know it is trained on millions of existing images, but when you say "... a bowl ..." in your prompt, how does it decide what the bowl should look like? Does it pick one of the bowls it's seen at random? It doesn't ever quite draw the same bowl twice, does it? Is it somehow "imagining" a bowl, the way a human would, and some all new image of a bowl pops into its "head"?
Ultimately, the link between words and their representations comes from the CLIP training. The model generates encodings (vectors) for both an image and its corresponding caption, and then the parameters of these encoders (the functions that generate the vectors) are tuned in order to minimize the angle between the textual and visual encodings that represent the same concept.
The core of your question is why minimizing the angle between like vectors is equivalent to learning what the "Platonic ideal" of a given object (in your example, a bowl) is, whether appearing as a textual representation or a visual one. This question is subtle and difficult to answer (if it's even a well-formulated question), but I'd say that the easiest interpretation is that the vector space is composed of a basis of vectors that each represent a distinct feature (which the model learns).
I find the concept of a GAN - a Generative Adversarial Network - useful.
My high-level attempt at explaining how those work is that you create two machine learning models, one that tries to create fake images and one that tries to see if an image is fake or not.
The first one says "here's an image", the second one says "that's a fake", the first one learns from that and tries again, then keep going until an image scores highly on the test.
The networks are adversarial because they are trying to outwit each other.
(I'm sure a ML researcher could provide a better explanation than I can, but that's the way I think about it.)
I don't believe Dall-E 2 incorporates GANs at all, but I haven't read the paper in detail. GANs were the best text-to-image models maybe a year ago but lately diffusion techniques are taking over.
It's not trained on labeled data so it doesn't know bowls are a specific concept necessarily. It's all statistical similarity in the same way Google Image Search works. (from the original CLIP paper, it seems to think an apple and the word "apple" written on a piece of paper are the same thing)
The model in step 3 produces an image encoding (something like a sketch of the output) from a text encoding (something like what you typed), and the unCLIP model in step 2 produces images from that encoding. How much variation you get inside a specific input word varies a lot and is spread across those models.
This could be misleading to some people. The original inputs to the CLIP encoders are pairs of images and text which are known to match.
Both the text encoder and the image encoder are then trained to minimize differences in output from eachother when given corresponding (“labeled” / “ground truth”) image/text pairs.
Oh that’s true, but I think they meant scraped collections like Wikipedia images with captions - so that is matching text, but it’s not labels written in a consistent vocabulary by annotator teams.
This is good and bad, since it makes it more robust.
I've always been skeptical of AI stuff (for obvious reasons/long-term implications), but I have to say this application has me excited beyond belief. This is pure magic. Kudos to the OpenAI team.
A few years ago when photorealistic facial image generation models started getting really good I had my first "holy crap" moment. OpenAI expanding the domain from faces to essentially anything is absolutely mind blowing. An absolutely seminal step forward undoubtedly!
Diffusion models seem like they're poised to completely replace GANs. They obviously work super well, and you don't have this super finicky minimax training problem.
Biggest problem for diffusion models were performance (as you need to iterate even at inference) But I'm not up to date with newest architectures maybe its already solved :P
I was wondering it if would be possible to train a neural network to do multiple iterative steps at once. As it turns out, it has already been done and it requires about 4 to 8 distilled iterations for comparable quality. If this pace keeps up, we will probably see similar running time to GANs in the near future.
Could be... Except their page (should you choose to believe it, of course) specifically addresses the advantages:
"""
"Advantages over Traditional GANs" : Thus, we observe that our model exhibits _better training stability_ and mode coverage.
"Why is Sampling from Denoising Diffusion Models so Slow?" : After training, we generate novel instances by sampling from noise and iteratively denoising it _in a few steps_ using our denoising diffusion GAN generator.
I guess we will figure that out quite soon, but does it matter that much? Your only job with DALL-E 2 is to prompt it properly so if you want a female scientist, just say it so. If it comes up with the "wrong" gender or ethnicity, then it takes a second to fix it, which would probably take a bit less than ranting about it on Twitter :)
I can't think of a way that would "fix" this that wouldn't also make it less useful overall. If people are looking for people being shot by police officers, they probably already have those stereotypes and thus expectations of the end product. You can argue that you want to insert a certain morality set in the process, but that to me sounds a hell of a lot scarier than the scientist emptying the dishwasher being a women in 60% of the pictures. Once you have the mechanism for morality bias, you also have people with the capacity to change the settings.
One way to dodge this and other issues related to depiction of human bodies is to trim the dataset such that humans are not generally recognizable as realistic humans in the output. It is also currently explicitly forbidden by OpenAI to share publicly realistic images of human faces generated by DE2.
Via LessWrong.com: [1]
>"One place where DE2 clearly falls down is in generating people. I generated an image for [four people playing poker in a dark room, with the table brightly lit by an ornate chandelier], and people didn't look human -- more like the typical GAN-style images where you can see the concept but the details are all wrong.
>Update: image removed because the guidelines specifically call out not sharing realistic human faces.
>Anything involving people, small defined objects, and so on, looks much more like the previous systems in this area. You can tell that it has all the concepts, but can't translate them into something realistic.
>This could be deliberate, for safety reasons -- realistic images of people are much more open to abuse than other things. Porn, deep fakes, violence, and so on are much more worrisome with people. They also mentioned that they scrubbed out lots of bad stuff from the training data; possibly one way they did that was removing most images with people.
>Things look much better with animals, and better again with an artistic style."
The authors of the Dall-E 2 / unCLIP paper describe some of their efforts to mitigate biases in the paper. ML models will always exhibit the biases present in their training dataset, without intervention. It's not really possible to remove bias from an ML model, at least not completely. Some stereotypes, but not all, are backed up by statistics. In those cases, should we completely remove the bias in the training dataset? Doing so would bias the model towards outputs that are not representative of the real world.
When people say that they want to remove bias from ML models, what they really mean is that they want to manipulate the output distribution into something they deem acceptable. I'm not arguing against this practise, there are plenty of situations where the output of an ML model is very clearly biased towards specific classes/samples. I'm merely arguing that there is no such thing as an unbiased model, just as there is no such thing as an unbiased human. Unbiased models would produce no output.
To get around some of these problems OpenAI restricted the training dataset (e.g. filtering sexual and violent content) and also prevent generating images with recognizable faces. This doesn't prevent bias but it does reduce the number of controversial outputs.
Can't really see anyone being shot (stereotype avoided), the dishwasher emptiers are male-ish presenting (stereotype confirmed?), and the nurses are female presenting (stereotype confirmed.)
It will, being a deterministic machine, generate any kind of wrongthink that is in its training data. Ironically, all of the media coverage of negative stereotypes by well intentioned activists probably even makes it more likely to generate this kind of data.
Great questions! I'd also be interested in this. I supposed the generations would mimic the general distribution of information that is on the internet, but what that would look like specifically is hard to say without OpenAI releasing more information.
Are we finally past the AI winter? We seem to be seeing major advances at least once a year. I recall there was a bit of a lull after GPT3, but clearly the boundaries of AI are expanding ridiculously fast.
GPT-3 was released 2 years ago, and in that time CLIP, GLIDE, and DALL-Es 1 and 2 have been released. All of this is just from OpenAI too! DL research is cranking along as quickly as ever imo!
You're going to be waiting a while, even now these models can't get the details right. Look at all the example pictures, everything is just wrong. You can obviously see what it is trying to get at, but it can't get there. Another example of the last 10% taking 90% of the time.
Jukebox. If you listen to Jukebox samples, recall that that was quite a while ago in dog/DL years, and imagine what the DALL-E 2 equivalent would be for a Jukebox 2...
I'm surprised no one has tried to launch a music generation startup based on Jukebox. I'd be interested in collaboration if anyone wants to work on it (and has compute resources).
I guess he's referring to the fact that the glut of investment around 2017-2018 was followed by disappointment due to startups overpromising. I agree that from the technical side (I mostly follow NLP, might be different in other subfields) there's been no hint of a winter.
Was the original AI winter with reference to AGI? I thought it was in reference to the resulting lack of research and interest after the "bubble bust". If we're not close to AGI now I can't imagine researchers 40 years ago really thought AGI was around the corner, right? Just curious, I'm not an expert on the history of ML!
I bet we're closer than most people think. Instruct GPT-3 can do semantic tasks just as efficiently as DALL-E 2 can draw. NLP tasks that took whole teams multiple years can be simply described in a few words and they work right away.
The entry barrier to implement new tasks will get very low. The large models will be the new operating system. This means more investments and data, leading to new improvements.
I believe GPT-3 is already close to median human level on most semantic tasks that fit in a 4000 token window. I'm researching how to use it right now for a variety of tasks, it just works from plain text requirements with no training.
But there's a quantum leap or two from (say) mindlessly producing comments that can occasionally fool readers on HN (as it has been used to do in the past) to it consciously joining in the conversation of its own volition and curiosity, then zoning out on Netflix while half-worrying about the future for GPT-4 jr and idly planning it's next server room refit.
It can't worry about jr because it doesn't have a dick. It's deficient in E's - embodied, embedded, enacted and extended. Our curiosities and values come from the game we play, so it would need to play the same game with us to develop them.
I think there have been several really, and they tend to follow hype periods, which over-promise.
I do think the last few years have been more productive than previous periods in advancing narrow AI, and to be fair to those researchers who just get on with the work, it is not on them if the advances are over-sold by others.
At least the big public display of this tech seems to me that it's mostly merging photos in interesting ways. That the 'seed' comes from a word is not hugely interesting to me.
I'm actually more curious if we could parse the underlying logic that ultimately it emulates to merge those images together.
It 'looks like' something kind of sophisticated is being modelled with AI but there's some nice algorithms hidden in there.
Luckily the use of Transformer models makes what's going on under the hood a bit more interpretable, but I think the fundamental part at which ideas are merged is translating from CLIP text embeddings to CLIP image embeddings.
The training principle of CLIP is very simple, but intuitively understanding how the diffusion prior maps between semantically similar textual and visual representations is a bit more unclear (if that's even a well-formulated question!)
It doesn’t merge images. It generates them from scratch. Sure it’s trained on a corpus of existing images, but I don’t think it „merges“ them any more than human artists do with images they have seen in their lifetime.
From one of the linked papers, it does both - generate from scratch (zero-shot), and edit existing template images.
> While our model can render a wide variety of text prompts zero-shot, it can can have difficulty producing realistic im ages for complex prompts. Therefore, we provide our model with editing capabilities in addition to zero-shot generation,
which allows humans to iteratively improve model samples until they match more complex prompts. Specifically, we fine-tune our model to perform image inpainting, finding that it is capable of making realistic edits to existing images using natural language prompts.
Unless that only applies to GLIDE and not to DAL-E?
Over the next decade, ML advancements will erode the monetary value of countless professions. Hopefully AI research will be turned towards solving the problems of society/economics/civilization before it is too late to avoid major disruptions in human wellbeing.
> "DALL-E 2's works very simply: ... a model called the prior maps the text encoding to a corresponding image encoding that captures the semantic information of the prompt contained in the text encoding. Finally, an image decoding model stochastically generates an image which is a visual manifestation of this semantic information."
> "The fundamental principles of training CLIP are quite simple: First, all images and their associated captions are passed through their respective encoders, mapping all objects into an m-dimensional space."
Not scared to admit I don't find this simple at all and I'm probably not in the target audience. I'd love a description that doesn't assume machine learning basics. Is there one?
it's "simple" because how it works is "just" brute-fucking-force. of course coming up with the architecture and making it fast (so it scales up well) is the challenge.
and scaling works .. because .. well, no one knows why (but likely because it's just a nice architecture for learning, evolution also converged on it without knowing why)
Nope, and you'll still need a pretty beefy computer to run the trained data. Currently GPT-NeoX-20B, the "open source GPT3," requires 42 GB of VRAM, so you're looking at minimum a $5-6k graphics card (though a Quadro RTX 8000 is actually in stock so there's that). Or use a service like GooseAI.
Eleuther.ai or some other open source / open research developers will likely try to reproduce DALL-E 2 but it'll take some time and a lot of donated hardware and cycles.
Maybe possible with a fabulous GPU, but still likely not, and if it did work it would take a horrendously long time. The real blocker is gonna be GPU memory. With an RTX 3090 you have 24 GB of GPU RAM and _might_ be able to try it, but I'm still not sure it would fit. The key model has 3.5 billion parameters, which at 16-bit requires 7GB of GPU-memory for each copy. Training requires 3 or 4 copies of the model, depending on the algorithm you use. And then you need memory for the data and activations, which you can reduce with a small batch size. But if it did fit, on a single GPU with a small batch size, you're probably looking at years of training time.
Sure, with a $5k card like that it would be physically capable, but still unreasonably slow. FWIW RTX 8000 is previous-generation - the current gen is the A6000, which is also $5k, and similarly spec'd but faster. If you're looking for a deep learning card, they're actually a great value - I've got a bunch of them.
I wonder whether it could be adapted to run on Apple M1 Ultra hardware, which can have 128GB of on-chip "unified" memory, can't find right away how much of that is available to the GPU or the neural cores, but if its even half of it, perhaps Apple will have the AI market cornered.
This is a cute question. Not today! I hope someone comes back to read this question in 10-15 years time, when we will all have the ability to train Dall-E quality models on our AR glasses.
Unfortunately, it is out of the question. OpenAI trains on hundreds of thousands of dollars of GPUs and even then the trainings take two weeks. Also, as far as I know their training data (400 M image/caption pairs) is not available to the public!
It is more like 10M+ for single run for the latest generation models[1]. This is a key reason why not lot of models are out there.
Few groups have that kind of money to commit, also the viability is not yet very clear , i.e. how much the model with make if commercialized so they can recoup the investment.
There is also cost of running the model on each API call, of course not factoring in any of the employee and other costs for sales marketing etc.
I've had this running locally for a while and whilst it's an alternative you can actually use, it's not anywhere near the quality of DALL-E, let alone DALL-E-2. (Although since I mainly use it for abstracts and inspiration, that's fine.)
If I have it generate a “bowl of soup” will I find an identical bowl in some clip art collection somewhere? How much does it deviate from the source images?
You can try to reverse image search - from what I've seen of other people doing this, the renditions are quite distinct. The diffusion process is ultimately the root of the model's ability to not just copy images. Variational methods truly allow for the learning of a distribution, which is why VAEs can generate new data and AEs can't.
Also, practically from a data point of view, the same object can be represented in numerous ways (different artistic styles, different filters, abstract paintings, etc.) and the model has to optimize across all of these samples. What this means is that the model truly is forced to learn the semantic meaning behind a concept and not just rely on specific features.
Check out the dropdown under the "Significance of CLIP to DALL-E 2" section in the article
I've played with tech like this for over a year now. You won't find the bowl in the source images. It doesn't evolve the noise into a source image. It slowly nudges the noise into feeling more and more bowl-like. Do that enough, and you get something that feels quite a bit like a bowl.
Put it this way: The model file is absurdly smaller than the half billion source images files. If it actually contained the source images, it would be the greatest feat of image compression ever. Instead it only contains the impression left over by the images. A lot closer to a memory than a jpg.
GPT-3 was sold with exclusive usage rights to MicroSoft, so maybe something along those lines with a different company (Meta?). As for what they will do with it, it's hard to say ...
You can use GPT-3 right now on OpenAI playground and there's commercial apps running on it that as far as I know aren't on Azure. It's not clear what they meant by exclusive.
"CLIP is trained on hundreds of millions of images and their associated captions..."
Does anyone have any insight as to which images were trained on? Was it all open-domain stuff? And if not were the original authors of those images made aware their work was being use to train an AI that would likely put them out of work? Were they compensated appropriately?
As far as I know, OpenAI has not made this dataset publicly available. IIRC the dataset is images scraped from instagram and their corresponding captions. Check out the CLIP paper for more details:
Theoretically, you could build a web-scraping tool to do something like this, but even storing that data would take an absolutely insane amount of storage.
I would assume OpenAI has some deal with Meta to make the creation of datasets like this easier.
Thanks for the link. I hope they do make the data set publicly available at some point so that the artists whose work helped train this can know.
I think, while it is absolutely impressive on a technical level what the OpenAI team has been able to do, it is also important to consider what damage it will do to artists and their livelihood.
Many professional artists stake their career on one unique style of art that they have honed and developed over many years. It's this unique style that clients generally pay for, and that now faces a very real threat of being stolen from them by a technology that frankly no human can hope to compete with. Without artist compensation, this can only lead to artists terminating their careers early once the AI has co-opted all work from them. Or future artists never beginning their careers in the first place. This is a net loss for humanity, as it will deprive us of works and styles of art that have yet to be imagined.
I'm not saying AI like this needs to go away. There is no putting that genie back in the bottle, of course. But it needs to be something that artists opt into. If someone's style is worth it for OpenAI to train on, then that style obviously should have a price tag. And it ought to be up to the artist whether they want to sell or not. Anything short of that is theft in my eyes.
Very unlikely they will share even if they really wished to, for one they don't want competition to have the same data, for another it will just expose them to lawsuits, some images are bound to be problematic and would slip their filters.
Openai just uses the word Open in their name, they are commercial company like any other, they are a business first.
To be honest I don't see how this does not lead to lawsuits anyway. Its only a matter of time before the AI spits out something that is a bit too close to an artist's actual work. Withholding their data, which undoubtedly contains copyrighted material if they were just blindly scraping, will only forestall things. The earlier they start being transparent the better it will be for them in the long run.
Lawsuits are cost of doing business in the U.S. They don't need to increase potential surface and invite them though.
Knowing that there is potentially copyright material does not give you standing to sue them, unless you can show reasonable cause that they could have used your copyrighted content no court will take it to discovery to prove it conclusively, your case will be thrown out for lack of standing.
However if they share their data set, then you can show that they actually took it and have standing to sue them. Making any potential case more complex and expensive.
Whoa, I guess that court precedent about legalizing web scraping has bigger ramifications for AI than for humans, I can't imagine Meta is very happy with having provided the training set for Microsoft's AI research (via OpenAI)
I've been generating little love stories between Putin and BoJo, things like going on romantic walks in the park and whatnot, and I cannot wait for DALL-E 2 access to add images to these :D
This is a truly horrifying piece of technology, destined to destroy the livelihoods of countless artists. It's incredible in terms of the technology, but... scary in equal measure.
I can't think of a single good reason for this to exist that doesn't have huge negative impacts on our world.
Why pay an artist/graphic designer when this does what you need?
"Now those damned creatives can go and find real jobs"
Last week at a birthday party, I met a 74-year-old career visual artist who still creates with various media: paint, colored pencils, sculpture, etc.
Curious for her thoughts on DALL-E, I pulled out my phone and invited her to generate some imagery. (I have early access via a family member at OpenAI.) She didn't skip a beat, and immediately started getting creative with it. We even did a "collaborative piece" à la Mad Lib.
I asked her if she felt threatened by DALL-E. Surprised by the question, she said: "No! I could see this really accelerating my process. Sometimes I'm blocked on an idea and I could see this being a great tool for finding inspiration. Can I get access to this?"
My take-away was that art is not zero-sum: someone's art isn't "less" because more entities are creating art. If computers can do it too — even if they're arguably more mechanical in the recombination of existing ideas (note: humans do the same) — nothing stops human art from being art.
Arguably, the biggest barrier to any creative domain is technical capability.
An immediate thought is that locked-in people who can only communicate by text would be able to share their thoughts more expressively.
In terms of the creation loop, anyone can create a bunch of AI-generated images. Wombo is huge right now. The differentiating factors will be prompt design, commitment to iteration, aesthetic-driven curation of generated works and presentation.
Photographers take and process thousands of photos to create just one masterpiece.
Art is zero sum in that there are a limited number of artist residencies, exhibitions and funds available.
In this case, we will likely see further contraction in the number of artists able to support themselves. There will ofc always be the super stars and hobbyists.
The amount of art that people want in their lives is much larger than the amount that's there now.
Artists who are willing to direct their talents towards satisfying others' desires for art will find the world is very positive sum. Those that vie for a limited number of spots in a prestige game may find that it's zero or even negative sum, but those are not good games to play anyways.
There is limited discretionary income available, why would I use $100 in funds to purchase digital art, when I can spend $0.10 to generate one and use the remaining $99.90 on a scuba diving experience?
Same was true many times throughout the history. Why do people still pay musicians to play in live concerts when they could listen to a recording? Why do people still watch other people play chess when they could watch two AIs play much better chess?
Think long term. Eventually AI will be able to do most of human jobs. As a result, products and services will become cheaper. As a result, people will have to work less for a living. As a result, more people will be able to draw and paint for pleasure, and not necessarily to make a buck.
> As a result, products and services will become cheaper. As a result, people will have to work less for a living.
I think we've seen this play out before and instead of reducing work, our standards of living increase and people keep working about the same amount. See e.g. the post industrial world where homemakers had to scrub clothes, then got machines to do the scrubbing, but subsequently had to clean the clothes more frequently.
We might be able to reduce the overall amount of human work only through extremely successful social/political reforms similar to the ones that outlawed child labor and established the 40 hour work week. Assuming the technology will cause it to happen is bound to lead to disappointment.
> Think long term. Eventually AI will be able to do most of human jobs. As a result, products and services will become cheaper. As a result, people will have to work less for a living. As a result, more people will be able to draw and paint for pleasure, and not necessarily to make a buck.
This is ahistorical. The fact is that you must at least seem to produce more market value than your total compensation in order for a company to hire you. There will simply be less people who make a "livable" wage while those who own these automations will become increasingly wealthy. Depending on how the market changes, there may also be increasing unemployment. But why would that matter? Unless unemployment gets too high, the market will continue to work as usual.
There's simply no reason for the owners and inheritors of an increasingly automated economy to share the value increase with their workers. The worker's wages will be market-determined just as before. Perhaps if unemployment gets too high it will be in their interests to offer something like UBI, though no reason for anything beyond what's strictly necessary for the economy to function, and the minimum required to avoid excessive social turmoil.
In the United States, the labor force participation rate for men has been in secular decline since 1950 (and perhaps earlier—-the Fred data only goes back to 1948):
I'm working on applying this technology to housing as we speak, you're very wrong IMHO.
Yeah some people are going to loose jobs over this, happens all the time. People are not isolated from the market, they function on it and need to take it into account.
Your argument is weak in that it could have been invoked for several previous inventions: ATMs replacing cashiers, search engines replacing librarians and so on.
I argue there is a difference because of the nature of the work. Machines aiding in farming is only a good thing, because it can maximize output and minimize input. People (largely) don't care about the process of how it was grown, but rather having the product to eat (Of course there's cruelty free agriculture, organic, etc. but stay with me here). But artistry is a personal thing, and maximizing the output of art pieces isn't something that most are interested in. Art is a uniquely unquantifiable subject, and we want it to have a personal and emotional connection to both the creator and the viewer, something that is lost when AI boil it down to it's essential components and rebuild them in it's image.
I think that this doesn't really help artists as much as just do it for them. Art, the way I see it, requires a human to do because it is something that requires emotion, something a robot could replicate but not feel. For example, a gut wrenching image of innocents being beat by police is gut wrenching because it's something that exists in the real world, and the artist and the subjects are real and their emotion is real. But a computer generated image only has a likeness; it doesn't have actual emotion.
I aldo don't think that it makes it "cheaper, more accessible, and allows more people to create". Digital art supplies being something readily available and relatively cheap to their classic counterparts is what makes things more accessible, and to make it more so would be to drive the cost down or something. Having the computer draw for you isn't exactly creating art.
And art isn't a commodity and I argue it shouldn't be a commodity. It's something, again, personal and special.
And this doesn't end at the visual arts, I think it applies too to writing. AI could write what's written in my journal word for word but my journal would have more value just by virtue of it being written by me.
I'm less worried about the jobs angle, as this can be viewed as a productivity tool. I'm more worried about the ability to use this tech for deep fakes. It's going to erode trust in society even further than it already has.
The cynic in me is wondering if that will make any difference. It's not like people need deep fakes or even the possibility of deep fakes to believe that the world is flat or that Obama was born in Kenya or that lizard people are running sex trafficking rings out the basements of pizza parlors with no basements.
People look at the objective reality, provided by the sources that should have the most credibility, and just shrug it off.
Right people should but this will only increase people being deluded, because of it's ease of use. And it's not like any of us are immune to being deluded either; I'm sure there are things I and others take as truths because the facts we found them upon were carefully fabricated to have no holes.
If I saw a masterfully crafted video of vaccines actually being implanted with microchips, wouldn't I believe it? I'm not an expert on identifying deepfakes, nor should I be just to consume media. I think this is a valid cause for concern and will make things worse rather than keep it the same.
I wouldn't believe a masterfully crafted video of vaccines actually being implanted with microchips unless the video were authenticated by at least one reputable news source. Provenance matters, and just like we don't believe extraordinary things based on single out-of-context photograph, we shouldn't believe extraordinary things based on a single out-of-context video.
> unless the video were authenticated by at least one reputable news source
You pretend as if the news actually bothers to corroborate everything it prints. Sometimes, they actively disengage from corroboration or critical thinking, particularly when it's favorable to their party or unfavorable to their party (ALL news sources are biased).
The news is also entirely corruptible. They already have been for some time.
If the incentives are there for the owners of the media to craft a fiction, or to support a fictional or exaggerated narrative, they will do it.
But even if some imaginary world existed where the media was actually incorruptible, suppose they got duped and ran a deep fake video as if it were real news. Human psychology is such that the video could still take root in the popular imagination and influence real world outcomes as a result. Even after being demonstrated of a video's inauthenticity.
I'm deeply worried that we don't have the proper psychological immune systems to weed out deeply fake audiovisual productions and prevent them from influencing our decision making or perception of reality.
you can't demand some technology not to be used when it is not a weapon
there isn't a reason to believe that our current world is in a stage that is free from changes, in fact our world become what it is due to invention of disruptive technologies, regardless you like it or not.
I suspect what's going on is that OpenAI has decoupled their PR activities from their science activities. They told the researchers to publish papers when they're ready, and then the PR apparatus decides when one is good enough to be crowned "DALL-E-2" and writes a blog post about it.