Error when trying to validate RAPIDS features

[VHolstein]

Dear RAPIDS developers,

I have started building my own RAPDIS container for an app that we use. For validation, I would like to compare GPS location features of the Barnett & Doryab pipelines. For testing, I have rearranged the location data into the same format as the aware_csv data. It seems to run fine, but I get this error message:
Warning message: The following named parsers don't match the column names: local_date_time, local_start_date_time, local_end_date_time

when running the job

rule phone_locations_processed_with_datetime:
    input: data/interim/test1/phone_locations_processed.csv, data/interim/time_segments/test1_time_segments.csv, data/external/participant_files/test1.yaml
    output: data/interim/test1/phone_locations_processed_with_datetime.csv, data/interim/test1/phone_locations_processed_with_datetime.done

and the output files are empty. GPS location data seems to look the same as the example csv for aware_csv data. The beginning of my config file is:

########################################################################################################################
#                                              GLOBAL CONFIGURATION                                                    #
########################################################################################################################

# See https://www.rapids.science/latest/setup/configuration/#participant-files
PIDS: [test1, test2, test3]

# See https://www.rapids.science/latest/setup/configuration/#automatic-creation-of-participant-files
CREATE_PARTICIPANT_FILES:
  CSV_FILE_PATH: "data/external/ReMAP_test_participants.csv" # see docs for required format
  PHONE_SECTION:
    ADD: True
    IGNORED_DEVICE_IDS: []
  FITBIT_SECTION:
    ADD: False
    IGNORED_DEVICE_IDS: []
  EMPATICA_SECTION:
    ADD: False

# See https://www.rapids.science/latest/setup/configuration/#time-segments
TIME_SEGMENTS: &time_segments
  TYPE: PERIODIC # FREQUENCY, PERIODIC, EVENT
  FILE: "data/external/timesegments_remap.csv"
  INCLUDE_PAST_PERIODIC_SEGMENTS: TRUE # Only relevant if TYPE=PERIODIC, see docs

# See https://www.rapids.science/latest/setup/configuration/#timezone-of-your-study
TIMEZONE:
  TYPE: SINGLE
  SINGLE:
    TZCODE: Europe/Berlin
  MULTIPLE:
    TZCODES_FILE: /spm-data/vault-data4/ReMAP/project_vincent/data/time_segments/multiple_timezones_example.csv
    IF_MISSING_TZCODE: STOP
    DEFAULT_TZCODE: Europe/Berlin
    FITBIT:
      ALLOW_MULTIPLE_TZ_PER_DEVICE: False
      INFER_FROM_SMARTPHONE_TZ: False

Would you have an idea where this error might be coming from? I was unable to locate the problem. If needed I can provide my entire config file or any other of the necessary helper files. Thanks a lot in advance!

[JulioV]

hi @VHolstein , that warning should not cause any issues. We have an open ticket to remove it.

A feature file could be empty for multiple reasons, for example, the dates in the participant file are filtering data out, there aren't enough rows of raw data to compute features, or more subtle bugs related to the data format or the feature provider configuraton params.

Can you reduce the problem to eight or so days of data for a single participant, a single time segment (daily for example), and a single location feature (locationvariance for example). And share your config file as well the test data (you can create random locations for sharing purposes). Alternatively, you can use our sample dataset to check what the input/output should look like.

[VHolstein]

I have tried processing again with your recommended setup, but it seems to be the same issue, though with a specific warning this time:

Error in make_style(x[["color"]]) : 
  Unknown style specification: br_magenta
Calls: readable_datetime ... <Anonymous> -> .Call -> format -> as_datetime -> Ops.POSIXt

In addition: Warning message:
The following named parsers don't match the column names: local_date_time, local_start_date_time, local_end_date_time 
Execution halted
[Thu Jan 19 23:42:14 2023]
Error in rule phone_locations_processed_with_datetime:
    jobid: 2
    output: data/interim/test1/phone_locations_processed_with_datetime.csv, data/interim/test1/phone_locations_processed_with_datetime.done

RuleException:
CalledProcessError in line 126 of /rapids/rules/preprocessing.smk:
Command 'set -euo pipefail;  Rscript --vanilla /rapids/.snakemake/scripts/tmp0_apkpch.readable_datetime.R' returned non-zero exit status 1.
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2339, in run_wrapper
  File "/rapids/rules/preprocessing.smk", line 126, in __rule_phone_locations_processed_with_datetime
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 560, in _callback
  File "/opt/conda/envs/rapids/lib/python3.7/concurrent/futures/thread.py", line 57, in run
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 546, in cached_or_run
  File "/opt/conda/envs/rapids/lib/python3.7/site-packages/snakemake/executors/__init__.py", line 2351, in run_wrapper
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message

I suspect that it has to do with the datetime specification somehow, but I don't understand it exactly. I have uploaded my config file, one participant with scrambled locations including the participant file and one location feature. and used the wifi, calls, message (etc) csv's from the example so that the analysis would run through. You can access the data here: https://github.com/VHolstein/fix_rapids_analysis if you want to look at it.