Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[TEST] upgrade dmlc-core #14510

Merged
merged 4 commits into from
Apr 13, 2019
Merged

Conversation

eric-haibin-lin
Copy link
Member

@eric-haibin-lin eric-haibin-lin commented Mar 24, 2019

Description

Upgrade the dmlc-core dependency to the latest commit.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Comments

Copy link
Member

@wkcn wkcn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM: )
Thank you for the update!

@eric-haibin-lin
Copy link
Member Author

eric-haibin-lin commented Mar 25, 2019

Looks like the CMakeFile in dmlc-core has been changed in multiple PRs that causes amalgamation to break:

@pinaraws
Copy link

@mxnet-label-bot add[pr-awaiting-merge]

@marcoabreu marcoabreu added the pr-awaiting-merge Review and CI is complete. Ready to Merge label Mar 25, 2019
@eric-haibin-lin eric-haibin-lin changed the title [DEP] upgrade dmlc-core [TEST] upgrade dmlc-core Mar 25, 2019
@wkcn wkcn added pr-work-in-progress PR is still work in progress and removed pr-awaiting-merge Review and CI is complete. Ready to Merge labels Mar 25, 2019
@hcho3
Copy link
Contributor

hcho3 commented Mar 25, 2019

@eric-haibin-lin Platform-specific headers such as Shlwapi.h should have been wrapped with platform-specific macros (_WIN32 etc). Does amalgamation not honor platform-specific macros?

@hcho3
Copy link
Contributor

hcho3 commented Apr 5, 2019

@eric-haibin-lin This patch should fix amalgamation:

diff --git a/amalgamation/amalgamation.py b/amalgamation/amalgamation.py
index 0a4be02b8..edf01941d 100644
--- a/amalgamation/amalgamation.py
+++ b/amalgamation/amalgamation.py
@@ -46,7 +46,13 @@ if platform.system() != 'Darwin':
 if platform.system() != 'Windows':
     blacklist.append('windows.h')
     blacklist.append('process.h')
+    blacklist.append('Shlwapi.h')

+if platform.system() == 'Windows':
+    blacklist.append('unistd.h')
+
+if 'freebsd' not in sys.platform:
+    blacklist.append('sys/endian.h')

 def get_sources(def_file):
     sources = []
@@ -94,6 +100,7 @@ def find_source(name, start, stage):

 re1 = re.compile('<([./a-zA-Z0-9_-]*)>')
 re2 = re.compile('"([./a-zA-Z0-9_-]*)"')
+re3 = re.compile('DMLC_EXECINFO_H')

 sysheaders = []
 history = set([])
@@ -129,6 +136,9 @@ def expand(x, pending, stage):
     with open(x, 'rb') as x_h:
         for line in x_h.readlines():
             uline = line.decode('utf-8')
+            if '#define DMLC_LOG_STACK_TRACE 1' in uline.strip():
+                # Do not enable stacktrace logging
+                continue
             if uline.find('#include') < 0:
                 out.write(line)
                 continue
@@ -138,10 +148,15 @@ def expand(x, pending, stage):
             m = re1.search(uline)
             if not m:
                 m = re2.search(uline)
-            if not m:
-                print(uline + ' not found')
-                continue
-            path = m.groups()[0]
+            if m:
+                path = m.groups()[0]
+            else:
+                m = re3.search(uline)
+                if m:
+                    path = 'execinfo.h'
+                else:
+                    print(uline + ' not found')
+                    continue
             h = path.strip('./') if "../3rdparty/" not in path else path
             if h.endswith('complex.h') and x.endswith('openblas_config.h'):
                 source = ''

@eric-haibin-lin
Copy link
Member Author

eric-haibin-lin commented Apr 6, 2019

Amalgamation test passed
Thank you @hcho3 !

@hcho3
Copy link
Contributor

hcho3 commented Apr 6, 2019

Somehow upgrading dmlc-core breaks Julia?

@eric-haibin-lin eric-haibin-lin merged commit 273ebc7 into apache:master Apr 13, 2019
haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019
* upgrade dmlc-core

* Update amalgamation.py
@eric-haibin-lin eric-haibin-lin deleted the dmlc-core branch September 2, 2019 23:35
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr-work-in-progress PR is still work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants