Skip to content

1.0.0

Choose a tag to compare

View all tags
@justinmerrell justinmerrell released this 03 Aug 15:28
· 1633 commits to main since this release
1.0.0
102592a
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

We're thrilled to announce the release of RunPod 1.0.0! After refining our CI/CD pipeline and gathering valuable feedback from thousands of users, we are ready to introduce significant enhancements and fixes.

New Features

Multi-Job Concurrency

Workers are now smarter and more efficient. They can fetch and process multiple jobs in parallel, accelerating your workflows and productivity. Here's what's new:

  • Flexible job fetching: You can now fine-tune a worker's operations. When starting a worker, pass a function into the concurrency_controller. This function determines the number of jobs a worker should fetch in parallel.

Job Streaming

Our platform offers a powerful streaming feature that allows users to receive real-time updates on job outputs. This is particularly useful when dealing with Language Model tasks. We support two types of streaming generator functions: regular generator and async generator.

Regular Generator Function:

def generator_streaming(job):
    for i in range(5):
        output = f"Generated token output {i}"
        yield output

Async Generator Function:

async def async_generator_streaming(job):
    for i in range(5):
        output = f"Generated async token output {i}"
        yield output
        await asyncio.sleep(1)  # Simulate an asynchronous task (e.g., LLM processing time).

Usage:

To enable streaming, use either a regular or async generator function to yield the output results. The generator will continuously produce token outputs, streamed to the client in real-time.

How to Stream:

To utilize the generator function for streaming, you must use the /stream/{jobid} endpoint. Clients will receive the streaming token outputs by making an HTTP request to this endpoint. The jobid parameter in the endpoint URL helps the server identify the specific job for which the streaming is requested.

Real-Time Updates:

Once the streaming is initiated, the /stream/{jobid} endpoint will continuously receive generated token outputs as the generator function yields them. This provides real-time updates to the client or user, ensuring they can access the latest results throughout the job execution.

Using generator-type handlers for streaming, our platform enhances the user experience by delivering dynamic and up-to-date information for tasks involving Language Models and beyond.

Updates & Bug Fixes

We've also made some crucial improvements and squashed a few bugs:

  • Improved Initialization: The worker implementation now leverages asyncio, significantly improving initialization times. Your workers are now ready to go faster than ever!
  • Cohesive File Naming: We've renamed some files to improve cohesion and understanding. Now it's even easier to understand the purpose of each file in the project.

What's Changed

New Contributors

Full Changelog: 0.10.0...1.0.0

Contributors

Jorghi12, justinmerrell, and 2 other contributors
Assets 2
Loading