How to use the output of test data from the split function

[OriginalGoku]

I have read the documentation on GluonTS a few times and I find it extremely helpful.
One issue that I am having a problem with is the split function.
In the documentation Splitting datasets into training and test the following code creates test data splits:

training_dataset, test_template = split(
    dataset, date=pd.Period("2015-04-07 00:00:00", freq="1H")
)
test_pairs = test_template.generate_instances(
    prediction_length=prediction_length,
    windows=3,
    distance=24,
)

but there is no documentation as to how to use the test_pairs for forecasting or evaluation.
test_pairs is a gluonts.dataset.split.TestData
When I use the following code, it runs without any error:
forecast_it, ts_it = make_evaluation_predictions(
dataset=test_pairs, predictor=predictor,
num_samples=100,
)
but then running the code: list(ts_it) will generate this error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-39-c943f8505fed>](https://localhost:8080/#) in <cell line: 1>()
----> 1 list(ts_it)

2 frames
[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in __iter__(self)
    384 
    385     def __iter__(self) -> Generator[Tuple[DataEntry, DataEntry], None, None]:
--> 386         yield from self.splitter.generate_test_pairs(
    387             dataset=self.dataset,
    388             prediction_length=self.prediction_length,

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in generate_test_pairs(self, dataset, prediction_length, windows, distance, max_history)
    249             for window in range(windows):
    250                 offset = window * distance
--> 251                 test = self.test_pair(
    252                     entry, prediction_length=prediction_length, offset=offset
    253                 )

[/usr/local/lib/python3.10/dist-packages/gluonts/dataset/split.py](https://localhost:8080/#) in test_pair(self, entry, prediction_length, offset)
    287         offset_ = self.offset + offset
    288         if self.offset < 0:
--> 289             offset_ += entry[FieldName.TARGET].shape[-1]
    290         assert (
    291             offset_ + prediction_length <= entry[FieldName.TARGET].shape[-1]

TypeError: tuple indices must be integers or slices, not str

I found a way around it by reconstructing a ListDataset as follows and then using the test_ds as input for the forecasting function:

k = np.array(list(test_pairs))
test_ds = ListDataset(
    [
        {
            FieldName.ITEM_ID: row['item_id'],
            FieldName.TARGET: row['target'],
            FieldName.START: row['start'],
            FieldName.FEAT_DYNAMIC_REAL: row['feat_dynamic_real']
        }
        for row in k[:,0]
    ],
    freq=freq,
)

In this method, I basically use the input part of the test_pairs to reconstruct a ListDataset. This seems to be a wasteful way to handle the TestData object.

[lostella]

@OriginalGoku the example in this PR is relevant: #2673 It shows how to use the output of split to evaluate forecasts accuracy.

This is a relatively new feature and may slightly change in future versions, so there’s yet no example in the documentation, but we should add it soon!

[OriginalGoku]

lostella
Thank you for your prompt reply. The example was very useful and it certainly makes things easier although an approach similar to what I used to solve my problem doesn't seem to me all that worst specially since my approach can still use the make_evaluation_predictions function.

One thing I am still not 100% sure is the way predict function handles test_data.
When using the "make_evaluation_predictions" function, we know that the function automatically "splits" the data in a way that the last prediction_length number of data points are going to be used for inference and the rest of the data will be used for forecasting the prediction_length datapoints.
For the new predict function, we are passing test_data.input as the data.
So does this mean that the predict function is different structurally from the make_evaluation_predictions and that the predict function uses all the data points to make the prediction?
I have read the code for the predict function and I assume this is the case but I'd like to make sure about this.

def predict(
        self,
        dataset: Dataset,
        num_samples: Optional[int] = None,
        num_workers: Optional[int] = None,
        num_prefetch: Optional[int] = None,
        **kwargs,
    ) -> Iterator[Forecast]:
        inference_data_loader = InferenceDataLoader(
            dataset,
            transform=self.input_transform,
            batch_size=self.batch_size,
            stack_fn=partial(batchify, ctx=self.ctx, dtype=self.dtype),
        )
        with mx.Context(self.ctx):
            yield from self.forecast_generator(
                inference_data_loader=inference_data_loader,
                prediction_net=self.prediction_net,
                input_names=self.input_names,
                output_transform=self.output_transform,
                num_samples=num_samples,

[lostella]

For the new predict function, we are passing test_data.input as the data.
So does this mean that the predict function is different structurally from the make_evaluation_predictions and that the predict function uses all the data points to make the prediction?

Yes, correct. Note that predict is not new, and it always worked this way.

Also note that the make_evaluation_predictions function precisely uses split, generate_instances and predict under the hood, see

gluonts/src/gluonts/evaluation/backtest.py

Lines 75 to 81 in 3ccb6d3

	_, test_template = split(dataset, offset=-window_length)
	test_data = test_template.generate_instances(window_length)
	return (
	predictor.predict(test_data.input, num_samples=num_samples),
	map(_to_dataframe, test_data),
	)

[OriginalGoku]

Excellent, this was very useful. Appreciate the time you took to reply. I will try to contribute back to this project hopefully once I understand it a little better.