Update arrow to latest master by pcmoritz · Pull Request #2100 · ray-project/ray

pcmoritz · 2018-05-19T02:41:14Z

This includes the following changes:

Serialization fix for Python 2 integers: apache/arrow#2055
Plasma performance improvements: apache/arrow#2056, apache/arrow#2015, apache/arrow#2022
PyTorch 0.4 Tensor serialization fix: apache/arrow#2033

AmplabJenkins · 2018-05-19T03:51:11Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5492/
Test PASSed.

AmplabJenkins · 2018-05-19T22:10:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5504/
Test PASSed.

robertnishihara · 2018-05-19T22:12:36Z

Looks like there are compilation errors on Mac

[ 44%] �[32mBuilding CXX object src/ray/CMakeFiles/ray_objlib.dir/object_manager/object_directory.cc.o�[0m

�[1m/Users/travis/build/robertnishihara/ray-private-travis/src/ray/object_manager/object_store_notification_manager.cc:19:60: �[0m�[0;1;31merror: �[0m�[1m'kDeprecatedPlasmaDefaultReleaseDelay' is deprecated: PLASMA_DEFAULT_RELEASE_DELAY is deprecated [-Werror,-Wdeprecated-declarations]�[0m

      store_client_.Connect(store_socket_name.c_str(), "", PLASMA_DEFAULT_RELEASE_DELAY));

�[0;1;32m                                                           ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:39:46: �[0m�[0;1;30mnote: �[0mexpanded from macro 'PLASMA_DEFAULT_RELEASE_DELAY'�[0m

#define PLASMA_DEFAULT_RELEASE_DELAY plasma::kDeprecatedPlasmaDefaultReleaseDelay

�[0;1;32m                                             ^

�[0m�[1m/Users/travis/build/robertnishihara/ray-private-travis/thirdparty/pkg/arrow/cpp/build/cpp-install/include/plasma/client.h:38:19: �[0m�[0;1;30mnote: �[0m'kDeprecatedPlasmaDefaultReleaseDelay' has been explicitly marked deprecated here�[0m

constexpr int64_t kDeprecatedPlasmaDefaultReleaseDelay = 64;

�[0;1;32m                  ^

�[0m1 error generated.

robertnishihara · 2018-05-19T22:13:51Z

Also, this looks related (it happened a couple times on private travis).

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.892s

FAILED (failures=1)

AmplabJenkins · 2018-05-20T21:56:06Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5519/
Test FAILed.

AmplabJenkins · 2018-05-20T22:18:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5518/
Test PASSed.

robertnishihara · 2018-05-20T23:29:19Z

There's still a linting error

ObjectStoreNotificationManager::ObjectStoreNotificationManager(
     boost::asio::io_service &io_service, const std::string &store_socket_name)
     : store_client_(), socket_(io_service) {
-  ARROW_CHECK_OK(
-      store_client_.Connect(store_socket_name.c_str(), "", plasma::kPlasmaDefaultReleaseDelay));
+  ARROW_CHECK_OK(store_client_.Connect(store_socket_name.c_str(), "",
+                                       plasma::kPlasmaDefaultReleaseDelay));

AmplabJenkins · 2018-05-20T23:34:41Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5522/
Test PASSed.

AmplabJenkins · 2018-05-21T00:37:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5528/
Test PASSed.

robertnishihara · 2018-05-21T03:18:11Z

Unfortunately this is still happening on Linux

======================================================================
FAIL: testLocalSchedulerFailed (__main__.ComponentFailureTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/component_failures_test.py", line 200, in testLocalSchedulerFailed
    True)
  File "test/component_failures_test.py", line 182, in check_components_alive
    self.assertTrue(component.poll() is None)
AssertionError: False is not true

----------------------------------------------------------------------
Ran 9 tests in 62.430s

FAILED (failures=1)

pcmoritz · 2018-05-22T01:03:52Z

The test failure is caused by this segfault I think:

Program received signal SIGSEGV, Segmentation fault.
std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
464	      : _M_t(__u.release(), std::forward<deleter_type>(__u.get_deleter())) { }
(gdb) bt
#0  std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >::unique_ptr(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __u=<optimized out>, this=0x800000000) at /usr/include/c++/5/bits/unique_ptr.h:464
#1  __gnu_cxx::new_allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (__p=0x800000000, this=0x2636cc0) at /usr/include/c++/5/ext/new_allocator.h:120
#2  std::allocator_traits<std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::construct<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >&, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >*, std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    __p=0x800000000, __a=...) at /usr/include/c++/5/bits/alloc_traits.h:530
#3  std::deque<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >, std::allocator<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > > >::emplace_back<std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> > >(std::unique_ptr<unsigned char [], std::default_delete<unsigned char []> >&&) (
    this=0x2636cc0) at /usr/include/c++/5/bits/deque.tcc:155
#4  plasma::PlasmaStore::push_notification (this=this@entry=0x2631d60, object_info=object_info@entry=0x2638f78)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:633
#5  0x000000000041124f in plasma::PlasmaStore::seal_object (this=this@entry=0x2631d60, object_id=..., 
    digest=digest@entry=0x7ffc5d1d39a0 "\001\253,\271cT", <incomplete sequence \352>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:439
#6  0x0000000000411a34 in plasma::PlasmaStore::process_message (this=0x2631d60, client=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:727
#7  0x0000000000413979 in plasma::PlasmaStore::<lambda(int)>::operator() (events=<optimized out>, __closure=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:524
#8  std::_Function_handler<void(int), plasma::PlasmaStore::connect_client(int)::<lambda(int)> >::_M_invoke(const std::_Any_data &, <unknown type in /home/ubuntu/ray/python/ray/core/src/plasma/plasma_store, CU 0x0, DIE 0x82032>) (__functor=..., __args#0=<optimized out>) at /usr/include/c++/5/functional:1871
#9  0x00000000004181da in std::function<void (int)>::operator()(int) const (__args#0=1, this=<optimized out>) at /usr/include/c++/5/functional:2267
#10 plasma::EventLoop::FileEventCallback (loop=<optimized out>, fd=<optimized out>, context=<optimized out>, events=<optimized out>)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:28
#11 0x0000000000433eb0 in aeProcessEvents (eventLoop=eventLoop@entry=0x2624cc0, flags=flags@entry=3)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:412
#12 0x00000000004342cb in aeMain (eventLoop=0x2624cc0) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/thirdparty/ae/ae.c:455
#13 0x0000000000418388 in plasma::EventLoop::Start (this=<optimized out>) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/events.cc:69
#14 0x000000000040d2a5 in plasma::PlasmaStoreRunner::Start (use_one_memory_mapped_file=false, hugepages_enabled=false, directory="P;\035]", system_memory=<optimized out>, 
    socket_name=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", this=0x2624c20) at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:785
#15 plasma::start_server (socket_name=socket_name@entry=0x7ffc5d1d51d4 "/tmp/plasma_store66283217", system_memory=<optimized out>, plasma_directory="/dev/shm", 
    hugepages_enabled=hugepages_enabled@entry=false, use_one_memory_mapped_file=use_one_memory_mapped_file@entry=false)
    at /home/ubuntu/ray/thirdparty/build/arrow/cpp/src/plasma/store.cc:820

AmplabJenkins · 2018-05-22T06:11:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5559/
Test FAILed.

AmplabJenkins · 2018-05-22T06:39:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5557/
Test PASSed.

AmplabJenkins · 2018-05-22T07:53:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5564/
Test FAILed.

AmplabJenkins · 2018-05-22T07:54:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5563/
Test FAILed.

AmplabJenkins · 2018-05-22T08:06:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5561/
Test PASSed.

AmplabJenkins · 2018-05-22T08:25:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5562/
Test PASSed.

AmplabJenkins · 2018-05-22T09:16:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5567/
Test PASSed.

AmplabJenkins · 2018-05-22T09:26:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5570/
Test FAILed.

AmplabJenkins · 2018-05-23T02:08:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5586/
Test PASSed.

AmplabJenkins · 2018-05-23T02:11:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5588/
Test FAILed.

AmplabJenkins · 2018-05-23T04:22:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5589/
Test FAILed.

pcmoritz · 2018-05-23T23:48:01Z

Jenkins retest this please

AmplabJenkins · 2018-05-24T01:00:42Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5605/
Test PASSed.

richardliaw · 2018-05-24T02:54:45Z

@pcmoritz tests look like they're passing?

* master: Prototype named actors. (ray-project#2129) Update arrow to latest master (ray-project#2100) [DataFrame] Speed up dtypes (ray-project#2118) do not fetch from dead Plasma Manager (ray-project#2116) [DataFrame] Refactor GroupBy Methods and Implement Reindex (ray-project#2101) Initial Support for Airspeed Velocity (ray-project#2113) Use automatic memory management in Redis modules. (ray-project#1797) [DataFrame] Test bugfixes (ray-project#2111) [DataFrame] Update initializations of IndexMetadata which use outdated APIs (ray-project#2103)

* fix-a3c-torch: Prototype named actors. (ray-project#2129) Update arrow to latest master (ray-project#2100) [DataFrame] Speed up dtypes (ray-project#2118) do not fetch from dead Plasma Manager (ray-project#2116)

Update arrow to latest master

f517a1b

pcmoritz mentioned this pull request May 19, 2018

Update to latest arrow? #2084

Closed

fix plasma manager test

fc8bc19

pcmoritz added 2 commits May 20, 2018 14:11

update

04245e9

linting

aa85320

pcmoritz added 2 commits May 20, 2018 15:26

fix

b9d7280

fix linting

c58e05c

pcmoritz added 2 commits May 21, 2018 22:26

debug code

fabe7ca

fix

22e8b65

pcmoritz added 4 commits May 21, 2018 23:43

update

881b9da

update

acddd4c

update

2e9c84e

update

96e813b

update

41b3aaf

fix

c4912fb

pcmoritz added 2 commits May 22, 2018 17:59

fix

34775a3

fix

4018f85

switch back to apache arrow repo

d0021c1

robertnishihara approved these changes May 24, 2018

View reviewed changes

robertnishihara merged commit 225608e into ray-project:master May 24, 2018

robertnishihara deleted the update-arrow branch May 24, 2018 07:26

pschafhalter mentioned this pull request May 24, 2018

Ray serialization turns ints into longs in Python 2. #1014

Closed

This was referenced May 25, 2018

Ray fails to serialize Torch tensor. #955

Closed

Fix A3C PyTorch implementation #2036

Merged

Comments

Conversation

pcmoritz commented May 19, 2018

Uh oh!

AmplabJenkins commented May 19, 2018

Uh oh!

AmplabJenkins commented May 19, 2018

Uh oh!

robertnishihara commented May 19, 2018

Uh oh!

robertnishihara commented May 19, 2018

Uh oh!

AmplabJenkins commented May 20, 2018

Uh oh!

AmplabJenkins commented May 20, 2018

Uh oh!

robertnishihara commented May 20, 2018

Uh oh!

AmplabJenkins commented May 20, 2018

Uh oh!

AmplabJenkins commented May 21, 2018

Uh oh!

robertnishihara commented May 21, 2018

Uh oh!

pcmoritz commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 22, 2018

Uh oh!

AmplabJenkins commented May 23, 2018

Uh oh!

AmplabJenkins commented May 23, 2018

Uh oh!

AmplabJenkins commented May 23, 2018

Uh oh!

pcmoritz commented May 23, 2018

Uh oh!

AmplabJenkins commented May 24, 2018

Uh oh!

richardliaw commented May 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants