[SUPPORT] OOM occurs a few days after the Flink jobmanager restarts

[1032851561]

Describe the problem you faced

A flink job writes to hudi table , but the jobmanager always has OOM after a period of time.

Application application_1669378398064_0078 failed 1 times (global limit =2; local limit is =1) due to AM Container for 
appattempt_1669378398064_0078_000001 exited with exitCode: -104

Failing this attempt.Diagnostics: [2022-12-08 13:19:49.156]Container 
[pid=39129,containerID=container_e07_1669378398064_0078_01_000001] is running beyond physical memory limits. Current usage: 
4.0 GB of 4 GB physical memory used; 7.0 GB of 8.4 GB virtual memory used. Killing container.

my table options:

  'read.streaming.enabled' = 'true',
  'path' = 'hdfs://xxx/hudi-warehouse/mytable',
  'hive_sync.enable' = 'true',
  'connector' = 'hudi',
  'read.streaming.check-interval' = '30',
  'hoodie.datasource.write.hive_style_partitioning' = 'true',
  'index.state.ttl' = '3650',
  'hoodie.datasource.write.partitionpath.field' = 'create_day',
  'changelog.enabled' = 'true',
  'clean.retain_commits' = '300',
  'table.type' = 'MERGE_ON_READ',
  'hive_sync.mode' = 'hms',

Expected behavior

A clear and concise description of what you expected to happen.

Environment Description

• Hudi version : 0.11.1

• Flink version : 0.14.5

My Questions

1. Is this normal?
2. Why does jobmanager use so much memory?
3. Should I increase memory or use other tuning methods

[yihua]

@1032851561 Thanks for raising the issue. Could you look at the Flink Web UI (better to take a screenshot and post it here) and see which tasks failed due to OOM?

[yihua]

@1032851561 Could you also share the stacktrace before this OOM happened? One possible root cause could be the Hive sync which is mentioned in #7561.

[1032851561]

@1032851561 Thanks for raising the issue. Could you look at the Flink Web UI (better to take a screenshot and post it here) and see which tasks failed due to OOM?

My flink job is only to write data of a table . From 2G of memory to 5G, the utilization rate of job manager is maintained at more than 90% until an error is reported. eg: Current usage:
4.0 GB of 4 GB physical memory used; 7.0 GB of 8.4 GB virtual memory used. Killing container.

[1032851561]

@1032851561 Could you also share the stacktrace before this OOM happened? One possible root cause could be the Hive sync which is mentioned in #7561.

Does hive sync occur in task manager ? My situation is job manager oom. I dump the stack and find that the most memory-consuming data structure is [B and [C . I guess it's caused by filesystemview ?

[conorzhao]

i also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library

[1032851561]

i also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library

Writing hudi jobs consumes a lot of jm memory, because timeline server, hive sync and compaction run in job manager. There was a hive sync bug that caused oom before version 0.11.

[conorzhao]

i also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library

Writing hudi jobs consumes a lot of jm memory, because timeline server, hive sync and compaction run in job manager. There was a hive sync bug that caused oom before version 0.11.

We have enabled offline compact and offline metadata synchronization, JM with 30GB of mem, but we still have frequent ooms，hudi version：0.13.1

[danny0405]

What version did you use, there are some hive sync bug that can cause OOM before 0.12.2 and 0.13.0, you can check this page for related bugs: https://www.yuque.com/yuzhao-my9fz/kb/flqll8#QmQk4

[1032851561]

i also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library

Writing hudi jobs consumes a lot of jm memory, because timeline server, hive sync and compaction run in job manager. There was a hive sync bug that caused oom before version 0.11.

We have enabled offline compact and offline metadata synchronization, JM with 30GB of mem, but we still have frequent ooms，hudi version：0.13.1

Every hudi table has a embed timeline server(a web server base on Javalin+jetty) by default, try to disable?

[conorzhao]

i also encountered the issue of OOM when Flink JM runs for a period of time. My usage scenario is to synchronize the entire library of Flink cdc MySQL to Hudi, with over 100 tables in one library

Writing hudi jobs consumes a lot of jm memory, because timeline server, hive sync and compaction run in job manager. There was a hive sync bug that caused oom before version 0.11.

We have enabled offline compact and offline metadata synchronization, JM with 30GB of mem, but we still have frequent ooms，hudi version：0.13.1

Every hudi table has a embed timeline server(a web server base on Javalin+jetty) by default, try to disable?

thx,i will try