AgentSkillsCN

runwayml

使用 Runway API 生成 AI 视频、图像和音频。在通过 Runway 从图像生成视频、将文本转换为视频、将视频转换为视频、进行角色表演、将文本转换为图像、将文本转换为语音、制作音效或进行语音处理时,均可使用该 API。

SKILL.md
--- frontmatter
name: runwayml
description: Generate AI videos, images, and audio with Runway API. Use when generating video from images, text-to-video, video-to-video, character performance, text-to-image, text-to-speech, sound effects, or voice processing with Runway.
license: MIT
compatibility: Requires internet access and a RunwayML API key (RUNWAYML_API_SECRET).
metadata:
  author: RunwayML
  version: 1.0.0

Runway API

Generate AI videos, images, and audio using Runway's API. Features Runway's latest Gen-4.5 model for high-quality text-to-video and image-to-video generation, plus Gen-4 variants and third-party models from Google (Veo, Gemini) and ElevenLabs.

Recommended: Use gen4.5 for best results - the newest and most capable video generation model.

Setup: See Installation Guide. API key goes in RUNWAYML_API_SECRET env var.

Quick Start

Python

python
from runwayml import RunwayML

client = RunwayML()

# Image-to-video with latest Gen-4.5 model
task = client.image_to_video.create(
    model="gen4.5",
    prompt_image="https://example.com/image.jpg",
    prompt_text="A timelapse on a sunny day with clouds flying by",
    ratio="1280:720",
    duration=10
).wait_for_task_output()

print(f"Video URL: {task.output[0]}")

# Text-to-video (no image required)
task = client.image_to_video.create(
    model="gen4.5",
    prompt_text="A serene mountain landscape at sunset with clouds drifting",
    ratio="1280:720",
    duration=10
).wait_for_task_output()

Node.js

javascript
import RunwayML from "@runwayml/sdk";

const client = new RunwayML();

// Image-to-video with latest Gen-4.5 model
const task = await client.imageToVideo
  .create({
    model: "gen4.5",
    promptImage: "https://example.com/image.jpg",
    promptText: "A timelapse on a sunny day with clouds flying by",
    ratio: "1280:720",
    duration: 10,
  })
  .waitForTaskOutput();

console.log(`Video URL: ${task.output[0]}`);

// Text-to-video (no image required)
const textTask = await client.imageToVideo
  .create({
    model: "gen4.5",
    promptText: "A serene mountain landscape at sunset",
    ratio: "1280:720",
    duration: 10,
  })
  .waitForTaskOutput();

cURL

bash
# Image-to-video
curl -X POST "https://api.dev.runwayml.com/v1/image_to_video" \
  -H "Authorization: Bearer $RUNWAYML_API_SECRET" \
  -H "X-Runway-Version: 2024-11-06" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gen4.5",
    "promptImage": "https://example.com/image.jpg",
    "promptText": "A timelapse on a sunny day",
    "ratio": "1280:720",
    "duration": 10
  }'

# Text-to-video (no image required)
curl -X POST "https://api.dev.runwayml.com/v1/image_to_video" \
  -H "Authorization: Bearer $RUNWAYML_API_SECRET" \
  -H "X-Runway-Version: 2024-11-06" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gen4.5",
    "promptText": "A serene mountain landscape at sunset",
    "ratio": "1280:720",
    "duration": 10
  }'

# Poll for result (use returned task id)
curl "https://api.dev.runwayml.com/v1/tasks/{task_id}" \
  -H "Authorization: Bearer $RUNWAYML_API_SECRET" \
  -H "X-Runway-Version: 2024-11-06"

All Available Models

Video Generation

ModelInputPricingUse Case
gen4.5Text or Image12 credits/secNewest & recommended - Best quality text/image-to-video (no audio)
gen4_turboImage5 credits/secFast image-to-video (no audio), iteration
gen4_alephVideo + Text/Image15 credits/secVideo-to-video transformation (no audio)
act_twoImage or Video5 credits/secCharacter performance/motion (no audio)
veo3Text or Image40 credits/secGoogle Veo high-quality video with audio
veo3.1Text or Image40 credits/secGoogle Veo 3.1 with keyframes & audio
veo3.1_fastText or Image15 credits/secGoogle Veo 3.1 faster/cheaper & audio

Image Generation

ModelInputPricingUse Case
gen4_imageText + References (optional)5 credits/720p, 8 credits/1080pHigh-quality with style transfer
gen4_image_turboText + References (required)2 credits/image (any res)Fast iteration
gemini_2.5_flashText + References5 credits/imageGoogle Gemini image gen

Audio Generation (ElevenLabs)

ModelInput → OutputPricing
eleven_multilingual_v2Text → Speech1 credit/50 chars
eleven_text_to_sound_v2Text → Sound Effects1 credit/6 sec
eleven_voice_isolationAudio → Clean Audio1 credit/6 sec
eleven_voice_dubbingAudio → Dubbed Audio1 credit/2 sec
eleven_multilingual_sts_v2Speech → Speech1 credit/2 sec

1 credit = $0.01. Get credits at dev.runwayml.com


Video Generation

Gen-4.5 (Text-to-Video and Image-to-Video)

The latest and most capable Runway model supporting both text-only and image-to-video generation.

Text-to-Video

Generate videos from text descriptions only:

python
task = client.image_to_video.create(
    model="gen4.5",
    prompt_text="A serene mountain lake at sunrise with mist rising from the water",
    ratio="1280:720",
    duration=10,
    seed=12345  # Optional: reproducibility
).wait_for_task_output()

Text-to-Video Aspect Ratios: Landscape 1280:720 (16:9) | Portrait 720:1280 (9:16)

Image-to-Video

Animate existing images with motion:

python
task = client.image_to_video.create(
    model="gen4.5",
    prompt_image="https://example.com/image.jpg",
    prompt_text="Camera slowly pushes in, leaves rustling in the breeze",
    ratio="1584:672",
    duration=10,
    seed=12345
).wait_for_task_output()

Image-to-Video Aspect Ratios:

  • Widescreen: 1280:720 (16:9), 1584:672 (21:9 ultra-wide)
  • Standard: 1104:832 (4:3)
  • Portrait: 720:1280 (9:16), 832:1104 (3:4)

Duration: 2-10 seconds | Pricing: 12 credits/second (60 credits minimum for 5 sec)

Gen-4 Turbo (Image-to-Video)

python
task = client.image_to_video.create(
    model="gen4_turbo",
    prompt_image="https://example.com/image.jpg",
    prompt_text="Camera slowly pushes in, leaves rustling",
    ratio="1280:720",
    duration=5,
    seed=12345           # Optional: reproducibility
).wait_for_task_output()

Aspect Ratios: Landscape 1280:720, 1584:672, 1104:832 | Portrait 720:1280, 832:1104 | Square 960:960

Aleph (Video-to-Video)

Transform existing videos with text/image guidance:

python
task = client.video_to_video.create(
    model="gen4_aleph",
    video_uri="https://example.com/source.mp4",
    prompt_text="Transform to anime style",
    references=[{"uri": "https://example.com/style_ref.jpg"}]  # Optional style reference
).wait_for_task_output()

Aspect Ratios: Adds 848:480 (landscape) and 480:848 (portrait) to Gen-4 options.

Act-Two (Character Performance)

Drive character motion from reference performance. Objects require type discriminators:

python
task = client.character_performance.create(
    model="act_two",
    character={"type": "image", "uri": "https://example.com/character.jpg"},  # or type: "video"
    reference={"type": "video", "uri": "https://example.com/performance.mp4"}
).wait_for_task_output()

Character types: image (character performs in static environment) or video (character performs with some of its own movement)

Veo (Google)

Google's Veo models for text-to-video and image-to-video. Veo models include audio generation - making them ideal when you need video with sound.

python
# Text-to-video with audio (no image required)
task = client.image_to_video.create(
    model="veo3.1",  # or "veo3", "veo3.1_fast"
    prompt_text="A cinematic shot of a rocket launching at sunset with roaring engines"
).wait_for_task_output()

# Image-to-video with audio
task = client.image_to_video.create(
    model="veo3.1",
    prompt_image="https://example.com/starting_frame.jpg",
    prompt_text="Smooth camera movement through the scene with ambient nature sounds"
).wait_for_task_output()

Note: Gen-4.5 and Gen-4 models produce silent video. Use Veo if you need audio, or add audio separately with ElevenLabs models (see Audio Generation section).


Image Generation

Gen-4 Image with References

Use reference images with @mention syntax in prompts:

python
# gen4_image - reference_images is optional
task = client.text_to_image.create(
    model="gen4_image",
    ratio="1920:1080",
    prompt_text="A beautiful mountain landscape at sunset"
).wait_for_task_output()

# With references
task = client.text_to_image.create(
    model="gen4_image",
    ratio="1920:1080",
    prompt_text="@EiffelTower painted in the style of @StarryNight",
    reference_images=[
        {"uri": "https://example.com/eiffel.jpg", "tag": "EiffelTower"},
        {"uri": "https://example.com/starry.jpg", "tag": "StarryNight"}
    ]
).wait_for_task_output()

Note: gen4_image_turbo requires reference_images (at least one). Use gen4_image for text-only generation.

Untagged references apply as general style:

python
reference_images=[
    {"uri": "https://example.com/subject.jpg", "tag": "subject"},
    {"uri": "https://example.com/style.jpg"}  # No tag = style reference
]

Audio Generation

ElevenLabs models for text-to-speech, sound effects, and voice processing.

Voice Presets: Maya, Arjun, Serene, Bernard, Billy, Mark, Clint, Mabel, Chad, Leslie, Eleanor, Elias, Elliot, Noah, Rachel, James, Katie, Tom, Wanda, Benjamin

Text-to-Speech

python
task = client.text_to_speech.create(
    model="eleven_multilingual_v2",
    prompt_text="Hello, welcome to RunwayML!",
    voice={"type": "runway-preset", "preset_id": "Maya"}
).wait_for_task_output()

Sound Effects

python
task = client.sound_effect.create(
    model="eleven_text_to_sound_v2",
    prompt_text="Thunder rumbling in the distance, rain on a window"
).wait_for_task_output()

Voice Isolation

python
task = client.voice_isolation.create(
    model="eleven_voice_isolation",
    audio_uri="https://example.com/noisy_audio.mp3"
).wait_for_task_output()

Voice Dubbing

python
task = client.voice_dubbing.create(
    model="eleven_voice_dubbing",
    audio_uri="https://example.com/speech.mp3",
    target_lang="es"  # Spanish
).wait_for_task_output()

Supported languages: en, hi, pt, zh, es, fr, de, ja, ar, ru, ko, id, it, nl, tr, pl, sv, fil, ms, ro, uk, el, cs, da, fi, bg, hr, sk, ta

Speech-to-Speech

Convert speech to a different voice. Requires type discriminators:

python
task = client.speech_to_speech.create(
    model="eleven_multilingual_sts_v2",
    media={"type": "audio", "uri": "https://example.com/original.mp3"},  # or type: "video"
    voice={"type": "runway-preset", "preset_id": "Maya"}
).wait_for_task_output()

Input Requirements

Size Limits

TypeURL LimitData URI LimitEphemeral Upload
Image16MB5MB (3.3MB pre-encoding)200MB
Video32MB16MB200MB
Audio32MB16MB200MB

Supported Formats

Images: JPEG, PNG, WebP (no GIF)

Videos: MP4 (H.264/H.265/AV1), MOV (ProRes), MKV, WebM

Audio: MP3, WAV, FLAC, M4A, AAC

Base64 Data URIs

python
import base64

with open("image.jpg", "rb") as f:
    data_uri = f"data:image/jpeg;base64,{base64.b64encode(f.read()).decode()}"

task = client.image_to_video.create(
    model="gen4_turbo",
    prompt_image=data_uri,
    prompt_text="Gentle movement"
).wait_for_task_output()

Task Management

All operations are async. Use wait_for_task_output() (polls automatically, 10 min timeout).

Statuses: PENDINGRUNNINGSUCCEEDED / FAILED / CANCELED

THROTTLED = rate-limited, treat as PENDING

Canceling/Deleting Tasks

python
# Cancel running task or delete completed task
client.tasks.delete(task.id)

See Task Management for manual polling and batch processing.


Prompting Tips

Gen-4 thrives on simplicity. Start simple, iterate.

  • Describe single scenes (5-10 sec clips)
  • Use clear physical descriptions, not conceptual language
  • Reference subjects generically: "the subject", "she"
  • Avoid negative phrasing - "no blur" produces unpredictable results

See Prompting Guide for camera movements and advanced techniques.


Error Handling

python
from runwayml import RunwayML, APIError, RateLimitError

client = RunwayML()

try:
    task = client.image_to_video.create(...).wait_for_task_output()
    if task.status == "FAILED":
        print(f"Generation failed: {task.failure}")
except RateLimitError:
    print("Rate limited - SDK retries automatically")
except APIError as e:
    print(f"API error {e.status_code}: {e.message}")
CodeMeaningAction
400Invalid inputFix request parameters
401Invalid API keyCheck RUNWAYML_API_SECRET
429Rate limitSDKs auto-retry with backoff
503Service unavailableSDKs auto-retry

Parameters Quick Reference

Video (gen4.5)

ParameterTypeOptions
modelstring"gen4.5", "gen4_turbo", "veo3", "veo3.1"
durationnumber2 to 10 (seconds)
ratiostringText-to-video: "1280:720" (16:9), "720:1280" (9:16)<br>Image-to-video adds: "1584:672" (21:9), "1104:832" (4:3), "832:1104" (3:4)
prompt_textstringMotion/scene description (required)
prompt_imagestringURL or base64 (optional for text-to-video, required for image-to-video)
seednumberOptional, for reproducibility

Image (gen4_image)

ParameterTypeOptions
modelstring"gen4_image", "gen4_image_turbo"
ratiostring"1920:1080", "1280:720", etc.
prompt_textstringImage description
reference_imagesarray[{"uri": "...", "tag": "..."}]

References

Official Documentation