Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark fetcher is now able to fetch event logs via REST API #225

Merged
merged 11 commits into from
Apr 12, 2017
Merged

Spark fetcher is now able to fetch event logs via REST API #225

merged 11 commits into from
Apr 12, 2017

Commits on Mar 21, 2017

  1. [WIP] Spark fetcher is now able to fetch event logs via REST API 

    This commit allows Dr. Elephant to fetch Spark logs without universal
read access to eventLog.dir on HDFS. SparkFetcher would use SparkRestClient
instead of SparkLogClient if configured as

    <params>
      <use_rest_for_eventlogs>true</use_rest_for_eventlogs>
    </params>

The default behaviour is to fetch the logs via SparkLogClient/WebHDFS.

Closes linkedin#199
    Sergei Lebedev committed Mar 21, 2017
    Configuration menu
    Copy the full SHA
    d406c3d View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2017

  1. Addressed the review comments 

    'SparkRestDerivedData' now has a dedicated optional field for storing
'SparkLogDerivedData' fetched via REST.
    Sergei Lebedev committed Mar 22, 2017
    Configuration menu
    Copy the full SHA
    6a10752 View commit details
    Browse the repository at this point in the history

  2. Removed unused imports

    Sergei Lebedev committed Mar 22, 2017
    Configuration menu
    Copy the full SHA
    c3fe15d View commit details
    Browse the repository at this point in the history

  3. Fixed compilation 

    by defaulting 'logDerivedData' to 'None' and 'fetchLogs' to 'false'
    Sergei Lebedev committed Mar 22, 2017
    Configuration menu
    Copy the full SHA
    e60f170 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2017

  1. Fixed compilation on JDK7

    Sergei Lebedev committed Mar 23, 2017
    Configuration menu
    Copy the full SHA
    b46433a View commit details
    Browse the repository at this point in the history

  2. Moved 'getLogData' closer to 'getApplicationLogs'

    Sergei Lebedev committed Mar 23, 2017
    Configuration menu
    Copy the full SHA
    84b50b6 View commit details
    Browse the repository at this point in the history

  3. Added a note on out-of-async vals

    Sergei Lebedev committed Mar 23, 2017
    Configuration menu
    Copy the full SHA
    00c2ef7 View commit details
    Browse the repository at this point in the history

  4. Added a test for REST-based log fetching

    Sergei Lebedev committed Mar 23, 2017
    Configuration menu
    Copy the full SHA
    ca7806d View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2017

  1. Added a client mode test for REST-based log fetching

    Sergei Lebedev committed Mar 24, 2017
    Configuration menu
    Copy the full SHA
    cac509b View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2017

  1. Make sure no logs are fetched if 'eventLog.enabled' is unset 

    Prior to this commit 'SparkRestClient' would ignore this 'SparkConf'
paremeter ant attempt to fetch the logs anyway.
    Sergei Lebedev committed Mar 25, 2017
    Configuration menu
    Copy the full SHA
    d06278b View commit details
    Browse the repository at this point in the history

Commits on Apr 12, 2017

  1. Reverted commented test in 'InfoExtractorTest'

    Sergei Lebedev committed Apr 12, 2017
    Configuration menu
    Copy the full SHA
    cdc0a6c View commit details
    Browse the repository at this point in the history