GEP 7 and updates to GEPs 1-5 necessitated by GEP 6

docs/geps.md

@@ -20,5 +20,6 @@ maxdepth: 1
 ../geps/gep-04
 ../geps/gep-05
 ../geps/gep-06
+../geps/gep-07
 ../geps/gep-x
 ```

docs/geps/gep-01.md

@@ -14,7 +14,7 @@
 - * Created
   * 2019-11-04
 - * Updated
-  * 2022-03-28
+  * 2025-07-XX
 - * Resolution
   * [Accepted](https://gettsim.zulipchat.com/#narrow/stream/309998-GEPs/topic/GEP.2001)
 ```
@@ -26,35 +26,17 @@ columns, parameters, Python identifiers (functions, variables), etc. should be n
 a nutshell and without explanations, these conventions are:
 
 1. Names follow standard Python conventions (`lowercase_with_underscores`).
-   Abbreviations of words that form a part of these names are always followed by an
-   underscore, unless it is the last word.
-
-1. Names should be long enough to be readable. However, we impose limits in order to
-   make GETTSIM usable in languages, which place limits on characters (Stata, in
-   particular).
-
-   - Column names that are typically user-facing have a hard limit of 20 characters.
-     These columns are documented in `DEFAULT_TARGETS` in `gettsim/config.py`.
-   - Other column names that users might potentially be interested in have a hard limit
-     of 32 characters.
-   - Columns geared at internal use (e.g., helper variables before applying a
-     favorability check) start with an underscore and there are no restrictions.
-     Internal variables should be used sparingly.
-
-1. If names need to be concatenated for making clear what a column name refers to (e.g.,
-   `arbeitslosengeld_2__vermögensfreibetrag_bg` vs.
-   `grundsicherung__im_alter__vermögensfreibetrag_eg`), the group (i.e., the tax or
-   transfer) that a variable refers to appears first.
-
-1. Because of the necessity of concatenated column names, there will be conflicts
-   between readability (1.) and variable length (2.). If such conflicts arise, they need
-   to be solved on a case by case basis. Consistency across different variants of a
-   variable names always has to be kept.
 
 1. The language should generally be English in all coding efforts and documentation.
    German should be used for all institutional features and directly corresponding
    names.
 
+1. The hierarchical naming convention (see {ref}`GEP 6 <gep-6>`) means that
+   abbreviations should be used only very sparingly.
+
+   An abbreviation is always followed by an underscore (unless it is the last word).
+   Underscores must not be used to separate German words that are pulled together.
+
 1. German identifiers use correct spelling even if it is non-ASCII (this mostly concerns
    the letters ä, ö, ü, ß).
 
@@ -93,32 +75,17 @@ in English. For column names, we always allow a pure ASCII option, see the next
 
 (gep-1-column-names)=
 
-## Column names (a.k.a. "variables" in Stata)
-
-We impose a hard limit of 20 characters for all column names that typically user-facing.
-This is for the benefit of Stata users, who face a strict limit of 32 characters for
-their column names. Furthermore, where developers using other languages may store
-different experiments in different variables, Stata users' only chance to distinguish
-them is to append characters to the column names.
-
-For the same reason, there is a hard limit of 32 characters for variables that users may
-reasonably request.
-
-If a column is only present for internal use, it starts with an underscore and there is
-no restriction on the number of characters. Internal columns should be used sparingly.
+## Column / Policy Function names (a.k.a. "variables" in Stata)
 
-Across variations that include the same identifier, this identifier should not be
-changed, even if it leads to long variable names (e.g., `kinderfreib`,
-`einkommensteuer__gesamteinkommen_y`). This makes searching for identifiers easier and
-less error-prone.
-
-If names need to be concatenated for making clear what a column name refers to (e.g.,
-`arbeitslosengeld_2__vermögensfreibetrag_bg` vs.
-`grundsicherung__im_alter__vermögensfreibetrag_eg`), the group (i.e., the tax or
-transfer) that a variable refers to appears first.
+The hierarchical naming convention (see {ref}`GEP 6 <gep-6>`) means that the
+highest-level identifier is the type of the programme (e.g., `einkommensteuer` or
+`kindergeld`). Very few variables live in the global namespace (e.g., the person
+identifier `p_id` or `alter`). A special case is the namespace `familie`, which lives in
+the global namespace.
 
 If a column has a reference to a time unit (i.e., any flow variable like earnings or
-transfers), a column is indicated by an underscore plus one of {`y`, `m`, `w`, `d`}.
+transfers), a column is indicated by an underscore plus one of {`y`, `q`, `m`, `w`,
+`d`}.
 
 The default unit a column refers to is an individual. In case of groupings of
 individuals, an underscore plus one of {`sn`, `hh`, `fg`, `bg`, `eg`, `ehe`} will
@@ -152,11 +119,6 @@ Note that households do not include flat shares etc.. Such broader definition ar
 currently not relevant in GETTSIM but may be added in the future (e.g., capping rules
 for costs of dwelling in SGB II depend on this).
 
-Open questions:
-
-- Can we use `arbeitslosengeld_2__bg_id` for both SGB II and SGB XII at the same time or
-  do we need to differentiate once we add serious support for SGB XII?
-
 Time unit identifiers always appear before unit identifiers (e.g.,
 `arbeitslosengeld_2__betrag_m_bg`).
 
@@ -168,12 +130,9 @@ general naming considerations here.
 - There is a hierarchical structure to these parameters in that each of them is
   associated with a group (e.g., `arbeitslosengeld`, `kinderzuschlag`). These groups or
   abbreviations thereof do not re-appear in the name of the parameter.
-- Parameter names should be generally be aligned with relevant column names. However,
-  since the group is not repeated for the parameter, it is often better not to
-  abbreviate them (e.g., `wohngeld_params["vermögensgrundfreibetrag"]` for the parameter
-  and `wohngeld__anspruchshöhe_m_wthh` for a column derived from it).
+- Parameter names should generally be aligned with relevant column names.
 
-## Other Python identifiers (Functions, Variables)
+## Other Python identifiers
 
 Python identifiers should generally be in English, unless they refer to a specific law
 or set of laws, which is where the same reasoning applies as above.
@@ -183,40 +142,15 @@ comprehension or a short loop, `i` might be an acceptable name for the running v
 A function that is used in many different places should have a descriptive name.
 
 The name of variables should reflect the content or meaning of the variable and not the
-type (i.e., float, int, dict, list, df, array ...). As for column names and parameters,
-in some cases it might be useful to append an underscore plus one of {`m`, `w`, `d`} to
-indicate the time unit and one of {`sn`, `hh`, `fg`, `bg`, `eg`, `ehe`} to indicate the
-unit of aggregation.
-
-## Examples
-
-As an example we can consider the naming of the parameter group `arbeitsl_geld`. The
-original name for this group of parameters was the abbreviation `alg`. This will seem
-like a suitable candidate for native speakers who are familiar with the German social
-security system; the abbreviation is commonly used to refer to this type of unemployment
-benefit. However, acronyms are generally not self-explanatory and users unfamiliar with
-them will thus not be able to guess their meaning without looking them up.
-
-More meaningful alternatives could be `alo_geld` or `arb_los_geld`. These names use
-abbreviations of the compounds of the term "Arbeitslosengeld", which the group name is
-supposed to reflect, and connect them in a Pythonic manner through underscores. However,
-`alo_geld` still leaves much room for interpretation and `arb_los_geld` separates the
-term "Arbeitslosen" in an odd way.
-
-The final choice `arbeitsl_geld` avoids all the disadvantages of the other options as it
-is an unambivalent, natural, and minimal abbreviation of the original term it is
-supposed to represent.
+type (i.e., float, int, dict, list, df, array ...).
 
 ## Alternatives
 
+- We worked with abbreviations before, but this hit limits and it led to never-ending
+  discussions (see `GEP-6 <gep-6>` for some history).
 - We considered using more English identifiers, but opted against it because of the lack
   of precision and uniqueness (see the example above: How to distinguish between
   Erziehungsgeld, Elterngeld, and Elterngeld Plus in English?).
-- Use one of the standards for column identifiers. They are not precise enough and
-  sometimes rather cryptic.
-- Do something like EUROMOD and include some hierarchy in column names (e.g. start with
-  `d_` for demographics). Should not be necessary if column names have clear enough
-  names. If anything, we would achieve this via a MultiIndex for the columns.
 
 ## A final note
 
@@ -249,6 +183,9 @@ for that. Quoting from there:
 
 ## Discussion
 
+The below refers to older versions of the GEP; it has been updated because
+`GEP-6 <gep-6>` made much of the original content obsolete.
+
 - GitHub PR: <https://github.com/iza-institute-of-labor-economics/gettsim/pull/60>
 - Discussion on provisional acceptance:
   <https://gettsim.zulipchat.com/#narrow/stream/309998-GEPs/topic/GEP.2001/near/189539859>
@@ -257,6 +194,8 @@ for that. Quoting from there:
 - GitHub PR for second update (concatenated column names, dealing with conflicting
   objectives, names for columns vs parameters):
   <https://github.com/iza-institute-of-labor-economics/gettsim/pull/342>
+- GitHub PR for third update (changes because of `GEP-6 <gep-6>`):
+  <https://github.com/iza-institute-of-labor-economics/gettsim/pull/855>
 
 ## Copyright
 

docs/geps/gep-02.md

@@ -11,13 +11,15 @@
   * Standards Track
 - * Created
   * 2022-03-28
+- * Updated
+  * 2025-07-xx
 - * Resolution
   * [Accepted](https://gettsim.zulipchat.com/#narrow/stream/309998-GEPs/topic/GEP.2002)
 ```
 
 ## Abstract
 
-This GEP lays out how GETTSIM stores the user-provided data (be it from the SOEP, EVS,
+This GEP lays out how GETTSIM stores user-provided data (be it from the SOEP, EVS,
 example individuals, ...) and passes it around to the functions calculating taxes and
 transfers.
 
@@ -26,20 +28,24 @@ in the data provided by the user (if it comes in the form of a DataFrame) or cal
 by GETTSIM. All these arrays have the same length. This length corresponds to the number
 of individuals. Functions operate on a single row of data.
 
-If a column name is `[x]_id` with `x` {math}`\in \{` `_hh`, `_bg`, `_fg`, `_ehe`, `_eg`,
-`_sn` {math}`\}`, it will be the same for all households, Bedarfsgemeinschaften, or any
+Arrays are stored in a nested dictionary (a pytree). One level of the dictionary is
+called a *namespace*. Its innermost level is called a *leaf name*. The data columns are
+called *leaves*.
+
+If a leaf name is `[x]_id` with `id` {math}`\in \{` `hh`, `bg`, `fg`, `ehe`, `eg`, `sn`,
+`wthh` {math}`\}`, it will be the same for all households, Bedarfsgemeinschaften, or any
 other grouping of individuals specified in {ref}`GEP 1 <gep-1-column-names>`.
 
-Any other column name ending in `_id` indicates a link to a different individual (e.g.,
-child-parent relations could be `parent_0_ind_id`, `parent_1_ind_id`; receiver of child
-benefits would be `kindergeldempf_id`).
+Any leaf name `p_id_[y]` indicates a link to a different individual (e.g., child-parent
+are specified via `(familie, p_id_elternteil_1)`, `(familie, p_id_elternteil_2)`; the
+recipient of child benefits would be `(kindergeld, p_id_empfänger)`).
 
 ## Motivation and Scope
 
 Taxes and transfers are calculated at different levels of aggregation: Individuals,
 couples, families, households. Sometimes, relations between individuals are important:
-parents and children, payors/receivers of alimony payments, which parent receives the
-`kindergeld` payments, etc..
+parents and children, payors/recipients of alimony payments, which parent receives
+Kindergeld payments, etc..
 
 Potentially, there are many ways of storing these data: Long form, wide form,
 collections of tables adhering to
@@ -54,7 +60,7 @@ N-dimensional arrays, etc.. As usual, everything involves trade-offs, for exampl
 - Almost all functions are much easier to implement when working with a single row. This
   is most important for the typical user and increasing the number of developers.
 
-- Modern tools for vectorization (e.g., Jax) scale best when working with single rows of
+- Modern tools for vectorization (e.g., JAX) scale best when working with single rows of
   data.
 
   Aggregation to groups of individuals (households, Bedarfsgemeinschaften,...) or
@@ -67,56 +73,45 @@ This is primarily internal, i.e., only relevant for developers as the highest-le
 interface can be easily adjusted. The default way to receive data will be one Pandas
 DataFrame.
 
-Users are affected only via the interface of lower-level functions. Under the proposed
-implementation, they will always work on single rows of data. Many alternatives would
-require users to write vectorised code, making filtering operations more cumbersome. For
-aggregation or referencing other individuals' data, GETTSIM will provide functions that
-allow abstracting from implementation details, see
-{ref}`below <gep-2-aggregation-functions>`.
+Users are affected only via the interface of lower-level functions. Functions will
+always work on single rows of data. Many alternatives would require users to write
+vectorised code, making filtering operations more cumbersome. For aggregation or
+referencing other individuals' data, GETTSIM will provide functions that allow
+abstracting from implementation details, see {ref}`below <gep-2-aggregation-functions>`.
 
 ## Detailed description
 
 The following discussion assumes that data is passed in as a Pandas DataFrame. It will
 be possible to pass data directly in the form that GETTSIM requires it internally. In
 that case, only the relevant steps apply.
 
-- GETTSIM will first make a check that all identifiers pointing to other individuals
-  (e.g., `kindergeldempf_id`) are valid.
-
-- GETTSIM will then create internal identifiers for individuals, households, and tax
-  units. GETTSIM will also generate appropriate columns with identifiers pointing to
-  other individuals. Columns with the original values are stored.
+- GETTSIM may make a check that all identifiers pointing to other individuals (e.g.,
+  `(kindergeld, p_id_empfänger)`) are valid.
 
-  All internal identifiers are integers starting at 0 and counting in increments of 1.
-  For individuals, they are sorted, implying they can be used to index into the arrays.
-  It also means that identifiers pointing to other individuals can be used directly for
-  indexing.
+- GETTSIM may make a check that there is no variation within a group of individuals if
+  the column name indicates that there must not be (e.g., all members sharing the same
+  `hh_id` must have the same `anzahl_personen_hh` in case the variable is provided as an
+  input column).
 
-  Because groups of individuals are not necessarily nested (e.g., joint taxation during
+- Because groups of individuals are not necessarily nested (e.g., joint taxation during
   separation phase but living in different households), they cannot be sorted in
-  general. In case users know their data allows sorting on all groups (i.e., all groups
-  have a nesting structure), they will be able to provide a `data_is_sorted` flag, which
-  defaults to `False`.
+  general.
 
 - The core of GETTSIM works with a collection of 1-d arrays, all of which have the same
   length as the number of individuals.
 
   These arrays form the nodes of its DAG computation engine (see {ref}`GEP 4 <gep-4>`).
 
-- GETTSIM returns an object of the same type and with the same identifiers that was
+- GETTSIM returns an object of the same type and with the same row identifiers that was
   passed by the user.
 
-- GETTSIM strives to show errors along with the original indices, but this may not
-  always be possible.
-
 (gep-2-aggregation-functions)=
 
 ### Grouped values and aggregation functions
 
 Often columns refer to groups of individuals. Such columns have a suffix indicating the
-group (see {ref}`GEP 1 <gep-1-column-names>`, currently `_hh`, `_bg`, `_fg`, `_ehe`,
-`_eg`, and `_sn`). These columns' values will be repeated for all individuals who form
-part of a group.
+group (see {ref}`GEP 1 <gep-1-column-names>`). These columns' values will be repeated
+for all individuals who form part of a group.
 
 By default, GETTSIM will check consistency on input columns in this respect. Users will
 be able to turn this check off.
@@ -132,20 +127,16 @@ Aggregation functions will be provided by GETTSIM.
 - As outlined in {ref}`GEP 4 <gep-4-aggregation-by-group-functions>` users will need to
   specify:
 
-  - The stringified name of the aggregated variable. This **must** end with a feasible
-    unit of aggregation, i.e., `_hh`, `_bg`, `_fg`, `_ehe`, `_eg`, or `_sn`
-  - The stringified name of the original variable.
-  - The type of aggregation {math}`\in \{` `sum`, `mean`, `max`, `min`, `any` {math}`\}`
+  - The name of the aggregated variable. This **must** end with a feasible unit of
+    aggregation, e.g., `_hh` or `_ehe`.
+  - The type of aggregation {math}`\in \{` `count`, `sum`, `mean`, `max`, `min`, `any`,
+    `all`, {math}`\}`
+  - The name of the original variable (not relevant for `count`)
 
   Note that as per {ref}`GEP 4 <gep-4-aggregation-by-group-functions>`, sums will be
   calculated implicitly if the graph contains a column `my_col` and an aggregate such as
   `my_col_hh` is requested somewhere.
 
-Note that the groups `tu` and `hh` may change in the future. Some might also be
-calculated via relations between household members, see
-[discussion](https://gettsim.zulipchat.com/#narrow/stream/224837-High-Level-Architecture/topic/Update.20Data.20Structures/near/180917151)
-on Zulip in this respect.
-
 ## Alternatives
 
 Versions 0.3 -- 0.4 of GETTSIM used a collection of pandas Series. This proved to be
@@ -157,13 +148,19 @@ households like
 \[here\](<https://www.tensorflow.org/api_docs/python/tf/math/segment_sum>) would have
 led to many merge-like operations in user functions.
 
+Versions 0.5 -- 0.7 of GETTSIM used flat collections of pandas Series. As the scope and
+detail of GETTSIM grew, maintaining uniqueness of column names across different areas of
+taxes and transfers became too difficult.
+
 ## Discussion
 
 - Some
   [discussion on Zulip](https://gettsim.zulipchat.com/#narrow/stream/224837-High-Level-Architecture/topic/Update.20Data.20Structures/near/180917151)
   re data structures.
 - Zulip stream for
   [GEP 2](https://gettsim.zulipchat.com/#narrow/stream/309998-GEPs/topic/GEP.2001/near/189539859).
+- GitHub PR for update (changes because of `GEP-6 <gep-6>`):
+  <https://github.com/iza-institute-of-labor-economics/gettsim/pull/855>
 
 ## Copyright