09_runwayml

Introduction to RunwayML

Session A: Generative Adversarial Networks, Semantic Maps

Objectives

• Understand the basics of how the GANs and Image Colorization models in RunwayML work
• Develop high-level understandings of nonlinear dimensionality reductions and the latent space.
• Understand how GANs can be applied to interactive systems to generate imagery.
• GAN Slides

Additional references

• Google's A.I. Experiments: Visualizing High-Dimensional Space
• Using Artificial Intelligence to Augment Human Intelligence by Shan Carter (see the first three sections for latent space descriptions)
• Octavio Good on Generative Adversarial Networks (21:54 - 28:35)
• GAN Lab by Minsuk Kahng, Nikhil Thorat, Polo Chau, Fernanda Viégas, and Martin Wattenberg
• In the Age of A.I., Is Seeing Still Believing? by Joshua Rothman

Creative GAN projects

• Playing a game of GANstruction by Helena Sarin (2019 Eyeo Talk]
• Misremembering and Mistranslating: GANs in Art Context and Fall of the House of Usher by Anna Ridler
• Using AI to Produce “Impossible” Tulips: Anna Ridler uses AI to bring “tulipmania” into the future by Elain Ayers
• Mario Klingemann’s Neurography: Cameraless Photography with Neural Networks
• Booksby.ai by Andreas Refsgaard and Mikkel Thybo Loose
• Unfinished by Roman Lipski
• Blackberry Winter by Christian Mio Loclair
• Meet AICAN, a machine that operates as an autonomous artist

Semantic Maps / Image Synthesis

• Semantic Image Synthesis with Spatially-Adaptive Normalization, original SPADE paper paper
• GAUGan online demo
• Learning to See by Memo Akten
• Uncanny Road by Anastasis Germanidis and Cristóbal Valenzuela
• AI Lab Workshop: Painting Landscapes with the Body

Session B: Model Training, Hosted Models and Networking

Objectives

• Learn how to integrate RunwayML with JavaScript and other software applications.
• Learn how to train your own StyleGAN model in RunwayML.
• Explore additional model training such as GPT-2 text generation and Object Detection.

Training Resources

• 🐭 Computer Mouse Conference GAN presentation and resources
• 📄 RunwayML Image Generation Model Training
• 📄 RunwayML Object Detection Model Training
• 🎥 Training StyleGAN machine learning models in Runway from Artificial Images
• 💻 Processing Cube Generator
• 🖼 Google Images Download
• 🖼 Flickr Scrape

Hosted Model Resources

• Hosted Models
• Interact with models in RunwayML over the network
• Networking examples and tutorials, organized by software application

Reading

• 📖 On Lacework: watching an entire machine-learning dataset by Everest Pipkin

Code Examples

• Glitch! RunwayML template - StyleGAN
• Glitch! RunwayML StyleGAN walk
• All Glitch + RunwayML templates

Assignment (Due Monday, Nov 9):

Choose one of the following options! You could also combine them by training your own model and connecting it to p5.js! Continue your reflection on RunwayML in a blog post. How is working with RunwayML from your code compared to the web interface? Include screenshots and screen captures of your workflow.

Model Training

• Collect your own image or text dataset and train a generative model in RunwayML. Render a latent walk video with the output (for images) or text output samples (for text).

RunwayML <-> p5.js

1. Create a p5.js sketch that receives data from RunwayML (using any model). You can use this glitch RunwayML template which hides the keys in a .env file. (If you work with the web editor only, be careful about leaving your token in your code and the model active in the RunwayML interface!)
2. Optionally, send data to RunwayML or use another programming environment or software tool besides p5.js.