Loading data from different files grouped by date

[OperationalFallacy]

Hi,

I'm developing a glue job that runs nightly and does some transforms for set of users (user1, user2 etc) every day.

Can glue load data from files grouped by day and user (or other identified), organized like this?

20.05.2020/user1/relation.json
20.05.2020/user1/accounts.json
20.05.2020/user1/profiles.json
20.05.2020/user2/relation.json
20.05.2020/user2/accounts.json
20.05.2020/user2/profiles.json

etc

I see that create_dynamic_frame_from_options can specify paths, but its not clear how to get data partitioned by identifier (user)? Or how to get dataframe by data type (relation, accounts etc).

df = glueContext.create_dynamic_frame_from_options("s3",
    {'paths': ["s3://bucket/20.05.2020"]},
     'recurs': true,
    format="json") \
    .toDF()

Could you give me some clues for glues data loads? :)

Thanks!

[OperationalFallacy]

Interesting that crawlers support this exact use case (https://docs.aws.amazon.com/glue/latest/dg/define-crawler.html#crawler-data-stores-exclude). However, is there a way to do the same in the glue job - if adding data to the catalog is not needed?

[moomindani]

Thank you for using AWS Glue.

You can run crawler against that kind of paths, and use the populated catalog with partition information to read data in Glue ETL job by using create_dynamic_frame_from_catalog.
https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-glue-context.html

BTW this repository is for hosting samples related to AWS Glue.
Your question does not depend on the samples.
I would like you to ask in AWS Forum or AWS Premium support.

Resolving.