Man with a Diffusion Model

When I tell people that I am making a movie in Unreal Engine, they often ask about AI. What effect will AI have on projects like this one? Am I worried about the implications of generative AI for filmmaking in general?

Right now, generative AI tools can't do most of what human actors and cinematographers can do. They also can't do most of what 3D applications like Blender, Substance Painter, or Unreal Engine can do. But, they are closing the distance. In the time it takes to finish this short movie, there will likely be AI tools that could help with certain elements of the production. I doubt there will be a silver-bullet AI that can replace all the software I am using. What about my next movie?

If I want to give the question of AI generated movies a fair shake, and try to draw out all the implications, I should imagine a world where these tools are as powerful as possible. For me, that means a user can speak or type in plain language and get exactly the movie they ask for. Users could make vague requests, like "give me a movie similar to the last five romantic comedies I watched," or they could make endlessly specific requests, like "let's increase the clouds in the sky by 50% and make the sunset redder. No, not that red. And, move the main character a couple centimeters to the left in the frame..."

I imagine most consumer requests would be on the vague end of the spectrum, and the meticulous prompting would be done by people trying to make something to call their own. With a tool this powerful, an imaginative storyteller could make a fresh, compelling movie. The question is, what kind of ecosystem would they be releasing that movie into? Who would watch it?

When a movie-watcher sits down in the evening, hoping to be entertained, an honorable few might request to browse the latest selection of films crafted from at least two thousand hours of human prompting. But, the average movie-watcher will say something like "show me Harry Potter 9 but put in more action scenes than you do for most people... and more sex scenes."

For a filmmaker (or any artist), this kind of frictionless creative power presents a conundrum. We are granted the ability to make any movie we ever dreamed of. We might immediately ask, what is the point?

Often, the movie itself is the point. For many creators, we just want to see the thing exist. Whether it finds an audience is a secondary concern. Still, it would be nice to get our movies in front of the people who would appreciate them. Would it be possible to earn a living, or gain status through our work?

To try and answer that question, I will ponder what the film world could look like if movie-genies were available to everyone. Because I want to focus on AI's implications for the human endeavor of filmmaking, my hypothetical scenario assumes that these genies will never act without a person prompting them, and that the only wishes they can grant involve images and sounds.

AI Stigma

In this future, a movie production that once demanded serious capital, large teams, and complex logistical coordination would be transformed into an effort more like writing a novel. It could be done by one person working at home during their free time, over the course of as many months or years as they pleased.

Since filmmaking would be more like writing a novel, we can look at the current world of self-published fiction for hints as to how the future of movies could play out. With services like Kindle Direct Publishing or Wattpad, the barriers for publishing a book have been low for years, long enough for an ecosystem to develop. The challenge for self-published authors is finding someone to choose their book from the ocean of books. These authors need to do their own marketing, generally through social media, to hopefully find an audience. Solo filmmakers in my imagined future would face the same obstacles.

In the writing world, there is a strong stigma against AI generated text. In today's film industry as well, AI is controversial. Would a movie made entirely with generative AI be shunned by most viewers?

I don't think it necessarily would. There is a philosophical difference between the work of a writer and the work of a movie director. Most people are not financially or physically prohibited from writing a good novel. The thinking is: if you can sit at your computer and type prompts to a Large Language Model, then nothing is stopping you from writing the thing yourself, other than your own lack of talent.

The situation for movies is different. Someone could be bursting with ideas for characters and scenes, but have no path to realizing that vision without AI tools. Movie lovers might accept this and show interest in the AI-assisted creations of inventive humans.

On typical movie sets, directors never do all the work themselves. In a behind-the-scenes featurette for Blade Runner, there is an anecdote about Ridley Scott asking the prop master to show him more and more coffee mugs so that he could pick the best one:

Directors like Ridley Scott get to prompt a talented cast and crew to provide dozens of options to sculpt into the final product. The director of an AI-generated movie would prompt their software in a similar fashion. The director could also push the AI tool farther than would be acceptable with a human crew member.

It is possible that the current stigma against AI is mainly due to the fact that these tools still aren't good enough. A talented author might be able to hand-hold a LLM to assemble a decent book, but someone looking for a weekend-read in their favorite genre still can't get a coherent novel from a simple prompt. If we cross that threshold, the landscape could shift. An influencer might share a video saying "Look, I know we hate AI, but... this book is wild." From there, AI genre fiction gets a foothold as a guilty pleasure, and we're on the slippery slope.

Likewise, as a thought experiment, pick a popular movie from our timeline. Let's say it's The Shawshank Redemption. That's a good movie. It's the highest rated movie on IMDB! Now, imagine that in an alternate future, for some reason, The Shawshank Redemption never existed. Then, someone makes it using generative AI. Maybe it takes one prompt, maybe it takes one million, doesn't matter. Are you really going to say The Shawshank Redemption is trash because it was made with AI? Be honest.

Entropy and Creation

This raises an interesting question: in the above scenario, how many instructions would it take to produce The Shawshank Redemption? I am assuming these AI tools will be as powerful as possible and trained on every movie ever made, including every movie that users prompt them to make. How likely would they be to produce a movie not found in their training data? Users could potentially be generating millions of movies a day. But, most users would give vague and unimaginative prompts. If we gathered all these millions of movies into a single online store, most of them would be variants of some common-denominator, mass-appeal movie.

It would not be The Shawshank Redemption. In fact, unless someone used a considerable number of prompts to make it, The Shawshank Redemption wouldn't even be on the store. This isn't because it would be impossible to produce a great movie from a brief prompt. It is simply because without extensive, deliberate instruction, the movie-genies would be unlikely to create any specific movie not found in their training set.

The reason I am making this point is to emphasize that, even if generative AI were capable of producing any image or sound imaginable, and even if it spewed out millions of movies a day, there would always be unexplored territory for filmmakers to venture into.

To illustrate this further, let's consider a movie that theoretically would be easier for AI to generate from a vague prompt: Christopher Nolan's The Dark Knight. Same as with Shawshank, for this thought experiment, The Dark Knight is absent from the AI's training data. But, the movie-genie has access to all the other Batman movies and TV series ever made. Let's give it access to all the Batman comics too, including the Frank Miller graphic novels. We even give it all of Nolan's other movies, including Batman Begins and The Dark Knight Rises. Then, we tell the movie-genie: Make the movie that comes between Batman Begins and The Dark Knight Rises. What are the odds that it will generate The Dark Knight? Close to zero. Granted, we might get a good Batman movie, one with mass appeal, but there is almost no chance it would resemble The Dark Knight released in 2008. The permutations of a two-and-a-half hour movie are too massive. Without The Dark Knight in its training data, how would the AI come up with Heath Ledger's portrayal of the Joker?

Storytellers looking to push the limits of their imagination will know that whatever AI gives them after only two or three prompts won't be good enough. If the movie-genie can satisfy your entire vision from a few sentences, that is proof you are in shallow waters and need to swim deeper.

Resurfacing

Let's say I go deep and spend hours every day for three years to create my masterpiece. Maybe it's the best movie ever made. There it is, on my computer's hard drive. How do I get people to watch it?

First, I send it to my friends and family. Most of them watch at least part of it. My mom watches all of it, and tells me it was really interesting. Unfortunately, none of my friends or family owns a movie theater. Now what?

Film festivals? Film festivals are already testing the generative AI waters. Many people oppose this, but I think an industry-wide boycott is unlikely, and while prestigious festivals may limit their showcase of AI-generated movies, I bet they will allow submissions. After all, if they charge $100 per feature submission, why wouldn't they let you submit the 50 movies your computer spat out over a weekend? If AI tools were as powerful and open as I am imagining, eventually the most innovative, trendy movies would come out of the gen-AI space. Would film festivals shift their programming to mostly AI-generated films? Regardless, same as today, getting into a major film festival without connections would be unlikely. I know my movie would be the best ever made, but Sundance would get a lot of competent AI submissions, and there is no accounting for taste.

There's always YouTube. Would YouTube still allow free uploads under an onslaught of AI-generated full-length movies? I don't know. Maybe, in a world where AI can spawn a feature film from just a few words of text, these bandwidth concerns wouldn't be an issue. Still, I don't want to just throw my movie on YouTube. I spent three years on this thing! There has to be someone willing to pay for it...

I imagine some kind of centralized marketplace for sharing or selling movies would spring up, but however I make my movie available, my next job would be to market it. There is a subreddit, r/selfpublish, where authors discuss advertising their books through social media and other means. The impression I get from r/selfpublish as well as other sources is that most self-published authors find success writing genre fiction, particularly romance, crime/mystery, and fantasy. Jane Friedman writes:

Some genres or categories of work are ideal for self-publishing efforts because the audience or market is already primed to consume things digitally and to discover their next read through online channels. Romance and erotica are prime examples.

But then there are other markets where you’ll find it very difficult to gain traction because they either haven’t moved predominantly to digital consumption, or the traditional publishers still perform a valued gatekeeping role—providing needed or wanted validation and curation. Literary work is one such market: you’ll find it hard to gain acceptance within a certain community unless your work has been editorially selected, plus the literary audience still prefers print.

It would probably be the same with the market for AI-generated movies. Amidst the romance and erotica, maybe some other visually expressive genres, like action/martial arts, would find more success than they do in the literary realm.

But, there is a complication. I am imagining a market supercharged with AI. Predictable genre pieces would be the first to get subsumed by the algorithm. The more people make them, the easier it will be for AI to one-shot them. Why pay for animal-human shapeshifter erotica if my movie-genie can make a dozen for free, tailored to my specifications?

Some readers admire certain writers, or enjoy the writer's community of fans, and want to support the writer. They will buy their favorite writer's new book, even if it is barely distinguishable from thousands of others. This is true in blogging as well. Most blogs offer material that readers could find in numerous other places, including ChatGPT, but readers subscribe because they like the author and they want to associate with other people who like the author.

So, I self-market my movie. After enough hours grinding on social media, I might be able to develop a following of enthusiasts willing to pay for my movie (but mainly they pay to access the community's private chat). Would it be enough to earn a living? Probably not. It could be a decent side-income, and I would have a tight community of movie lovers to share ideas with. Our subculture could carry on for a time, off the radar.

What if my film really is brilliant and ends up on everyone's radar? There is a specter looming. The fans would have their own movie-genies and would want to imitate and remix my movie. This could continue until derivatives of my film are so ubiquitous that it becomes a meme and suffers the same fate as the genre-films. Generative AI could mimic the film's style or narrative beats with a simple prompt. At that point, most people would be watching the imitations.

In this situation, making my own sequel or adaptation would feel pointless, because the market would already be saturated. My true fans might buy it either way, but I would feel compelled to give them something new. To make a splash, I would need to create new visual styles, new story structures, new character dynamics.

From a purely creative standpoint, this isn't necessarily a bad world. In today's industry, making a vapid, derivative movie can still cost millions of dollars and years of work. With all-powerful generative AI, imitation becomes cheap and easy, but innovation is still possible. As soon as a movie is sucked into the algorithms of the general public, consider it dead. Filmmakers must constantly break new ground to have a hope of being recognized.

The Studios

There have been self-published authors who, after growing a sizeable fan base, get a book deal with a traditional publisher looking to capitalize on a built-in audience. The publisher's goal is to manufacture a cultural event that diehard fans as well as newcomers will buy into. If a filmmaker in the gen-AI community grew popular enough, they might be offered a distribution deal by a movie studio or theater chain hoping to fill theaters across the country with people like this:

A theatrical distribution deal could be a chance for a solo filmmaker to cash in before their creation is memed into oblivion.

What else are the movie studios doing in this ecosystem? So far, I assumed a magical world, where generative AI tools run on personal computers and users are free to train their AI on the data of every movie, old and new. I avoided thorny issues like copyright.

In the totally open world, studios would probably still make prestige films with popular directors and actors, focused on theatrical releases. These traditional productions could have smaller budgets than today's blockbusters and emphasize their human performers.

Another possible scenario would be less open. If generative-AI technology requires too much compute for personal devices to handle and can only be offered by a large tech company, studios would have more leverage to protect their intellectual property. Studios might sue to prevent the AI company from training on their films, or at least prevent the AI from responding to prompts that refer to copyrighted material.

Studios could also license their content to the AI company. If Universal sold the rights to Back to the Future, consumers would be allowed to prompt their movie-genie for Back to the Future 4, or 5, or whatever, as long as those generated movies were locked inside their personal device.

It is difficult to envision all the particular ways the less-open scenario could play out. It would largely depend on the technical specifics of how the AI tools work.

Horseless Carriages

There is a good chance that, like the guy who asks for Harry Potter 9, I haven't pushed my imagination far enough. I pictured a world more or less like ours, only with free and abundant movie making. In actuality, these tools could lead us down unexpected paths.

If the barriers to movie making were blown open, corporations might just abandon 2D storytelling to the proles and focus their capital on an entirely new medium. Photography became widely accessible around the 1880s. Industry and technology pushed photography to the extreme and birthed film and television. If fast, cheap audiovisual creation were available on personal computers, what would that technology look like pushed to the limits that only large companies could afford?

We don't know when the current surge in AI capability will stabilize. It could stall well before the scenario I described here. In that case, every way that AI tools fall short of the movie-genie will be an additional opportunity for humans to apply their creativity. Or, if AI exceeds the movie-genie and becomes an agentic everything-genie... well, I don't think we'll be chatting about moving pictures.