Skip to content

Conversation

@jhamon
Copy link
Collaborator

@jhamon jhamon commented Nov 4, 2025

Add create_namespace method to Index and IndexAsyncio

Summary

This PR adds the create_namespace method to both synchronous and asynchronous Index clients, as well as the GRPC implementation. The method allows users to create namespaces in serverless indexes with optional schema configuration.

Changes

REST API Implementation (Sync & Async)

  • Request Factory (pinecone/db_data/resources/sync/namespace_request_factory.py):

    • Added CreateNamespaceArgs TypedDict
    • Added create_namespace_args method with validation for namespace name and optional schema handling
  • Resource Classes:

    • NamespaceResource.create() - Synchronous implementation
    • NamespaceResourceAsyncio.create() - Asynchronous implementation
    • Both methods accept name and optional schema (as dictionary) parameters
  • Interface Definitions:

    • Added create_namespace() abstract method to IndexInterface
    • Added create_namespace() abstract method to IndexAsyncioInterface
    • Both include comprehensive RST docstrings with examples
  • Class Implementations:

    • Index.create_namespace() - Delegates to namespace resource
    • IndexAsyncio.create_namespace() - Delegates to namespace resource with async support

GRPC Implementation

  • GRPCIndex (pinecone/grpc/index_grpc.py):
    • Added create_namespace() method with async_req support for GRPC futures
    • Handles schema conversion from dictionary to MetadataSchema proto object
    • Supports both synchronous and asynchronous (future-based) execution

Testing

  • Unit Tests (tests/unit_grpc/test_grpc_index_namespace.py):

    • test_create_namespace - Basic functionality
    • test_create_namespace_with_timeout - Timeout handling
    • test_create_namespace_with_schema - Schema conversion validation
  • Integration Tests (tests/integration/data/test_namespace.py):

    • test_create_namespace - Successful namespace creation
    • test_create_namespace_duplicate - Error handling for duplicate namespaces
  • Integration Tests (tests/integration/data_asyncio/test_namespace_asyncio.py):

    • test_create_namespace - Async successful namespace creation
    • test_create_namespace_duplicate - Async error handling for duplicate namespaces
  • GRPC Futures Integration Tests (tests/integration/data_grpc_futures/test_namespace_future.py):

    • test_create_namespace_future - Creating namespace with async_req=True
    • test_create_namespace_future_duplicate - Error handling with futures
    • test_create_namespace_future_multiple - Concurrent namespace creation

API Design

The create_namespace method signature is consistent across all implementations:

def create_namespace(
    self, 
    name: str, 
    schema: Optional[Dict[str, Any]] = None, 
    **kwargs
) -> NamespaceDescription
  • Public API: Uses Optional[Dict[str, Any]] for schema to avoid exposing OpenAPI types
  • Schema Format: Accepts a dictionary with fields key containing field definitions
  • Returns: NamespaceDescription object containing namespace information

Examples

REST API (Synchronous)

from pinecone import Pinecone

pc = Pinecone()
index = pc.Index(host="example-index.svc.pinecone.io")

# Create namespace without schema
namespace = index.create_namespace(name="my-namespace")

# Create namespace with schema
schema = {
    "fields": {
        "field1": {"filterable": True},
        "field2": {"filterable": False}
    }
}
namespace = index.create_namespace(name="my-namespace", schema=schema)

REST API (Asynchronous)

import asyncio
from pinecone import Pinecone

async def main():
    pc = Pinecone()
    async with pc.IndexAsyncio(host="example-index.svc.pinecone.io") as index:
        namespace = await index.create_namespace(name="my-namespace")
        print(f"Created namespace: {namespace.name}")

asyncio.run(main())

GRPC (Synchronous)

from pinecone.grpc import PineconeGRPC

pc = PineconeGRPC()
index = pc.Index(host="example-index.svc.pinecone.io")

namespace = index.create_namespace(name="my-namespace")

GRPC (Asynchronous/Futures)

from pinecone.grpc import PineconeGRPC
from concurrent.futures import as_completed

pc = PineconeGRPC()
index = pc.Index(host="example-index.svc.pinecone.io")

# Create namespace asynchronously
future = index.create_namespace(name="my-namespace", async_req=True)
namespace = future.result(timeout=30)

# Create multiple namespaces concurrently
futures = [
    index.create_namespace(name=f"ns-{i}", async_req=True) 
    for i in range(3)
]
for future in as_completed(futures):
    namespace = future.result()
    print(f"Created: {namespace.name}")

Type Hints

  • Public-facing methods use Optional[Dict[str, Any]] for schema parameter
  • Internal resource methods handle conversion from dict to OpenAPI models
  • GRPC implementation converts dict to MetadataSchema proto object

Error Handling

  • Validates that namespace name is a non-empty string
  • Raises PineconeApiException for REST API errors
  • Raises PineconeException for GRPC errors
  • Properly handles duplicate namespace creation attempts

Documentation

All methods include comprehensive RST docstrings with:

  • Parameter descriptions
  • Return value descriptions
  • Usage examples
  • Links to relevant documentation

Testing Status

✅ All unit tests passing
✅ All integration tests passing (REST sync/async)
✅ All GRPC futures integration tests passing

Notes

  • This operation is only supported for serverless indexes
  • Namespaces must have unique names within an index
  • Schema configuration is optional and can be added when creating the namespace or later

@jhamon jhamon changed the base branch from main to release-candidate/2025-10 November 4, 2025 11:36
@jhamon jhamon force-pushed the jhamon/create_namespace branch from 01dc0fe to 6b555d5 Compare November 4, 2025 11:54
@jhamon jhamon marked this pull request as ready for review November 4, 2025 16:25
@jhamon jhamon merged commit 23dda74 into release-candidate/2025-10 Nov 4, 2025
34 checks passed
@jhamon jhamon deleted the jhamon/create_namespace branch November 4, 2025 16:25
jhamon added a commit that referenced this pull request Nov 18, 2025
⚠️ **Python 3.9 is no longer supported.** The SDK now requires Python 3.10 or later. Python 3.9 reached end-of-life on October 2, 2025. Users must upgrade to Python 3.10+ to continue using the SDK.

⚠️ **Namespace parameter default behavior changed.** The SDK no longer applies default values for the `namespace` parameter in GRPC methods. When `namespace=None`, the parameter is omitted from requests, allowing the API to handle namespace defaults appropriately. This change affects `upsert_from_dataframe` methods in GRPC clients. The API is moving toward `"__default__"` as the default namespace value, and this change ensures the SDK doesn't override API defaults.

Note: The official SDK package was renamed last year from `pinecone-client` to `pinecone` beginning in version 5.1.0.  Please remove `pinecone-client` from your project dependencies and add `pinecone` instead to get  the latest updates if upgrading from earlier versions.

You can now configure dedicated read nodes for your serverless indexes, giving you more control over query performance and capacity planning. By default, serverless indexes use OnDemand read capacity, which automatically scales based on demand. With dedicated read capacity, you can allocate specific read nodes with manual scaling control.

**Create an index with dedicated read capacity:**

```python
from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    AwsRegion,
    Metric
)

pc = Pinecone()

pc.create_index(
    name='my-index',
    dimension=1536,
    metric=Metric.COSINE,
    spec=ServerlessSpec(
        cloud=CloudProvider.AWS,
        region=AwsRegion.US_EAST_1,
        read_capacity={
            "mode": "Dedicated",
            "dedicated": {
                "node_type": "t1",
                "scaling": "Manual",
                "manual": {
                    "shards": 2,
                    "replicas": 2
                }
            }
        }
    )
)
```

**Configure read capacity on an existing index:**

You can switch between OnDemand and Dedicated modes, or adjust the number of shards and replicas for dedicated read capacity:

```python
from pinecone import Pinecone

pc = Pinecone()

pc.configure_index(
    name='my-index',
    read_capacity={"mode": "OnDemand"}
)

pc.configure_index(
    name='my-index',
    read_capacity={
        "mode": "Dedicated",
        "dedicated": {
            "node_type": "t1",
            "scaling": "Manual",
            "manual": {
                "shards": 3,
                "replicas": 2
            }
        }
    }
)

pc.configure_index(
    name='my-index',
    read_capacity={
        "mode": "Dedicated",
        "dedicated": {
            "node_type": "t1",
            "scaling": "Manual",
            "manual": {
                "shards": 4,
                "replicas": 3
            }
        }
    }
)
```

When you change read capacity configuration, the index will transition to the new configuration. You can use `describe_index` to check the status of the transition.

See [PR #528](#528) for details.

You can now fetch vectors using metadata filters instead of vector IDs. This is especially useful when you need to retrieve vectors based on their metadata properties.

```python
from pinecone import Pinecone

pc = Pinecone()
index = pc.Index(host="your-index-host")

response = index.fetch_by_metadata(
    filter={'genre': {'$in': ['comedy', 'drama']}, 'year': {'$eq': 2019}},
    namespace='my_namespace',
    limit=50
)
print(f"Found {len(response.vectors)} vectors")

for vec_id, vector in response.vectors.items():
    print(f"ID: {vec_id}, Metadata: {vector.metadata}")
```

**Pagination support:**

When fetching large numbers of vectors, you can use pagination tokens to retrieve results in batches:

```python
response = index.fetch_by_metadata(
    filter={'status': 'active'},
    limit=100
)

if response.pagination and response.pagination.next:
    next_response = index.fetch_by_metadata(
        filter={'status': 'active'},
        pagination_token=response.pagination.next,
        limit=100
    )
```

The update method used to require a vector id to be passed, but now you have the option to pass a metadata filter instead. This is useful for bulk metadata updates across many vectors.

There is also a dry_run option that allows you to preview the number of vectors that would be changed by the update before performing the operation.

```python
from pinecone import Pinecone

pc = Pinecone()
index = pc.Index(host="your-index-host")

response = index.update(
    set_metadata={'status': 'active'},
    filter={'genre': {'$eq': 'drama'}},
    dry_run=True
)
print(f"Would update {response.matched_records} vectors")

response = index.update(
    set_metadata={'status': 'active'},
    filter={'genre': {'$eq': 'drama'}}
)
```

A new `FilterBuilder` utility class provides a type-safe, fluent interface for constructing metadata filters. While perhaps a bit verbose, it can help prevent common errors like misspelled operator names and provides better IDE support.

When you chain `.build()` onto the `FilterBuilder` it will emit a python dictionary representing the filter. Methods that take metadata filters as arguments will continue to accept dictionaries as before.

```python
from pinecone import Pinecone, FilterBuilder

pc = Pinecone()
index = pc.Index(host="your-index-host")

filter1 = FilterBuilder().eq("genre", "drama").build()

filter2 = (FilterBuilder().eq("genre", "drama") &
           FilterBuilder().gt("year", 2020)).build()

filter3 = (FilterBuilder().eq("genre", "comedy") |
           FilterBuilder().eq("genre", "drama")).build()

filter4 = ((FilterBuilder().eq("genre", "drama") &
            FilterBuilder().gte("year", 2020)) |
           (FilterBuilder().eq("genre", "comedy") &
            FilterBuilder().lt("year", 2000))).build()

response = index.fetch_by_metadata(filter=filter2, limit=50)

index.update(
    set_metadata={'status': 'archived'},
    filter=filter3
)
```

The FilterBuilder supports all Pinecone filter operators: `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in_`, `nin`, and `exists`. Compound expressions are build with and `&` and or `|`.

See [PR #529](#529) for `fetch_by_metadata`, [PR #544](#544) for `update()` with filter, and [PR #531](#531) for FilterBuilder.

You can now create namespaces in serverless indexes directly from the SDK:

```python
from pinecone import Pinecone

pc = Pinecone()
index = pc.Index(host="your-index-host")

namespace = index.create_namespace(name="my-namespace")
print(f"Created namespace: {namespace.name}, Vector count: {namespace.vector_count}")

namespace = index.create_namespace(
    name="my-namespace",
    schema={
        "fields": {
            "genre": {"filterable": True},
            "year": {"filterable": True}
        }
    }
)
```

**Note:** This operation is not supported for pod-based indexes.

See [PR #532](#532) for details.

For sparse indexes with integrated embedding configured to use the `pinecone-sparse-english-v0` model, you can now specify which terms must be present in search results:

```python
from pinecone import Pinecone, SearchQuery

pc = Pinecone()
index = pc.Index(host="your-index-host")

response = index.search(
    namespace="my-namespace",
    query=SearchQuery(
        inputs={"text": "Apple corporation"},
        top_k=10,
        match_terms={
            "strategy": "all",
            "terms": ["apple", "corporation"]
        }
    )
)
```

The `match_terms` parameter ensures that all specified terms must be present in the text of each search hit. Terms are normalized and tokenized before matching, and order does not matter.

See [PR #530](#530) for details.

**Update API keys, projects, and organizations:**

```python
from pinecone import Admin

admin = Admin() # Auth with PINECONE_CLIENT_ID and PINECONE_CLIENT_SECRET

api_key = admin.api_key.update(
    api_key_id='my-api-key-id',
    name='updated-api-key-name',
    roles=['ProjectEditor', 'DataPlaneEditor']
)

project = admin.project.update(
    project_id='my-project-id',
    name='updated-project-name',
    max_pods=10,
    force_encryption_with_cmek=True
)

organization = admin.organization.update(
    organization_id='my-org-id',
    name='updated-organization-name'
)
```

**Delete organizations:**

```python
from pinecone import Admin

admin = Admin()

admin.organization.delete(organization_id='my-org-id')
```

See [PR #527](#527) and [PR #543](#543) for details.

You can now configure which metadata fields are filterable when creating serverless indexes. This helps optimize performance by only indexing metadata fields that you plan to use for filtering:

```python
from pinecone import (
    Pinecone,
    ServerlessSpec,
    CloudProvider,
    AwsRegion,
    Metric
)

pc = Pinecone()

pc.create_index(
    name='my-index',
    dimension=1536,
    metric=Metric.COSINE,
    spec=ServerlessSpec(
        cloud=CloudProvider.AWS,
        region=AwsRegion.US_EAST_1,
        schema={
            "genre": {"filterable": True},
            "year": {"filterable": True},
            "rating": {"filterable": True}
        }
    )
)
```

When using schemas, only fields marked as `filterable: True` in the schema can be used in metadata filters.

See [PR #528](#528) for details.

The SDK now exposes header information from API responses. This information is available in response objects via the `_response_info` attribute and can be useful for debugging and monitoring.

```python
from pinecone import Pinecone

pc = Pinecone()
index = pc.Index(host="your-index-host")

response = index.query(
    vector=[0.1, 0.2, 0.3, ...],
    top_k=10,
    namespace='my_namespace'
)

for k, v in response._response_info.get('raw_headers').items():
    print(f"{k}: {v}")
```

See [PR #539](#539) for details.

We've replaced Python's standard library `json` module with `orjson`, a fast JSON library written in Rust. This provides significant performance improvements for both serialization and deserialization of request payloads:

- **Serialization (dumps)**: 10-23x faster depending on payload size
- **Deserialization (loads)**: 4-7x faster depending on payload size

These improvements are especially beneficial for:
- High-throughput applications making many API calls
- Applications handling large vector payloads
- Real-time applications where latency matters

No code changes are required - the API remains the same, and you'll automatically benefit from these performance improvements.

See [PR #556](#556) for details.

We've optimized gRPC response parsing by replacing `json_format.MessageToDict` with direct protobuf field access. This optimization provides approximately 2x faster response parsing for gRPC operations.

Special thanks to [@yorickvP](https://github.com/yorickvP) for surfacing the `json_format.MessageToDict` refactor opportunity. While we didn't merge the specific PR, yorick's insight led us to implement a similar optimization that significantly improves gRPC performance.

See [PR #553](#553) for details.

- **Type hints and IDE support**: Comprehensive type hints throughout the SDK improve IDE autocomplete and type checking. The SDK now uses Python 3.10+ type syntax throughout.
- **Documentation**: Updated docstrings with RST formatting and code examples for better developer experience.
- **Dependency updates**: Updated protobuf to 5.29.5 to address security vulnerabilities. Updated `pinecone-plugin-assistant` to version 3.0.1.
- **Build system**: Migrated from poetry to uv for faster dependency management.

- [@yorickvP](https://github.com/yorickvP) - Thanks for surfacing the gRPC response parsing optimization opportunity!
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants