Fix derivation of price_ref from PRICE_EMISSION

[macflo8]

Code sample or context

Trying to add and calibrate MACRO with an existing MESSAGEix-GLOBIOM scenario without exogenous input for price_ref, demand_ref and cost_ref:

import ixmp
import message_ix

from message_ix import Scenario
from sdmx.model import Annotation, Code

from message_ix_models.model import macro
from message_ix_models.util import identify_nodes

mp = ixmp.Platform("ixmp_dev")
scenario = Scenario(mp, model_name, scenario_name)
scenario = scenario.clone("MESSAGEix-Materials", "refactor_macro_calib")
scenario.set_as_default()

def _c(id, sector):
    return Code(id=id, annotations=[Annotation(id="macro-sector", text=sector)])

commodities = [
    _c("i_therm", "i_therm"),
    _c("i_spec", "i_spec"),
    _c("rc_spec", "rc_spec"),
    _c("rc_therm", "rc_therm"),
    _c("transport", "transport"),
]
context.model.regions = identify_nodes(scenario)
data = dict(
    config=macro.generate("config", context, commodities),
    aeei=macro.generate("aeei", context, commodities, value=0.02),
    drate=macro.generate("drate", context, commodities, value=0.05),
    depr=macro.generate("depr", context, commodities, value=0.05),
    lotol=macro.generate("lotol", context, commodities, value=0.05),
)
# Load other MACRO data from file
data2 = macro.load(private_data_path("macro", "SSP1"))
data.update(data2)

scenario.add_macro(data, check_convergence=False)

Expected result

Execution of the above code without errors.

Problem description

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\maczek\Anaconda3\envs\message_env\Scripts\mix-models.exe\__main__.py", line 7, in <module>
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\click\decorators.py", line 45, in new_func
    return f(get_current_context().obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\PycharmProjects\message-ix-models\message_ix_models\model\material\__init__.py", line 853, in test_calib
    scen = scenario.add_macro(data, check_convergence=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\PycharmProjects\message_ix\message_ix\core.py", line 723, in add_macro
    add_model_data(self, clone, data)
  File "C:\Users\maczek\PycharmProjects\message_ix\message_ix\macro.py", line 590, in add_model_data
    c2.get(k)
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\genno\core\computer.py", line 647, in get
    raise ComputationError(exc) from None
genno.core.exceptions.ComputationError: computing 'price_ref' using:

(<function extrapolate at 0x000001CF371B4D60>, <PRICE_COMMODITY:n-c-l-y-h:macro>, 'mapping_macro_sector', 'ym1')

Use Computer.describe(...) to trace the computation.

Computation traceback:
  File "C:\Users\maczek\PycharmProjects\message_ix\message_ix\macro.py", line 297, in extrapolate
    model_data.groupby(list(groupby_cols)).apply(fitted_intercept).rename("value")
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\pandas\core\groupby\groupby.py", line 1780, in apply
    return self._python_apply_general(f, self._obj_with_exclusions)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\pandas\core\groupby\groupby.py", line 1819, in _python_apply_general
    values, mutated = self.grouper.apply_groupwise(f, data, self.axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\pandas\core\groupby\ops.py", line 911, in apply_groupwise
    res = f(group)
          ^^^^^^^^
  File "C:\Users\maczek\PycharmProjects\message_ix\message_ix\macro.py", line 291, in fitted_intercept
    (b, m), _ = curve_fit(f, df["year"] - ym1, df["value"])
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\scipy\optimize\_minpack_py.py", line 977, in curve_fit
    res = leastsq(func, p0, Dfun=jac, full_output=1, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\maczek\Anaconda3\envs\message_env\Lib\site-packages\scipy\optimize\_minpack_py.py", line 424, in leastsq
    raise TypeError(f"Improper input: func input vector length N={n} must"
TypeError: Improper input: func input vector length N=2 must not exceed func output vector length M=1

Versions

Output of message-ix show-versions

ixmp:        3.8.1.dev3+g8a2aa91
     8a2aa91 (HEAD -> main, origin/main, origin/HEAD) Update minimum version of genno (#514)
message_ix:  3.8.1.dev2+g693f227.d20240122
     693f227 (HEAD -> main) Update minimum version of genno (#783)
message_ix_models: 2023.11.25.dev20+gaf092b1c.d20240122
     ea92d38a (HEAD -> fix/materials-W23, origin/fix/materials-W23) Add macro calibration CLI function
message_data: 2020.6.21.dev2788+gc6da04d1
     7a24436bf (HEAD -> materials_2023_move2, origin/materials_2023_move2) Reorder material imports

click:       8.1.7
dask:        2024.4.1
genno:       installed
graphviz:    0.20.1
jpype:       1.5.0
… JVM path:  C:\Users\maczek\Anaconda3\envs\message_env\Library\bin\server\jvm.dll
openpyxl:    3.1.2
pandas:      2.1.4
pint:        0.23
xarray:      2024.1.0
yaml:        6.0.1

iam_units:   installed
jupyter:     1.0.0
matplotlib:  3.7.0
plotnine:    0.12.4
pyam:        2.2.0

GAMS:        38.3.0

python:      3.11.7 | packaged by Anaconda, Inc. | (main, Dec 15 2023, 18:05:47) [MSC v.1916 64 bit (AMD64)]
python-bits: 64
OS:          Windows
OS-release:  10
machine:     AMD64
processor:   Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder:   little
LC_ALL:      None
LANG:        None
LOCALE:      ('English_United States', '1252')

[khaeru]

To add some more detail from our Slack conversation:

• This occurs in the specific case where the set of MESSAGE commodity codes with level="useful" include ‘extra’ codes (in this case, "non-comm") that do not have corresponding MACRO sector codes.
• This results in values of PRICE_EMISSION corresponding to those ‘extra’ commodities to be retained, when they should instead be dropped.
• @macflo8 tested a fix that adds an explicit filter on commodity; this appears to work.

To resolve the issue, we:

• Insert this fix.
• Check that existing tests pass.
• Add additional tests for this case.
• Adjust or update documentation as necessary.