Performance not scaling with multithreading

[abhijithch]

While running an example from recommender system package for movielens data, the performance of multithreaded parallel version does not seem to scale well.

To run the example download the dataset from here, unzip the files to any folder. Clone this repository and run the movielens example in julia built with multithreading,

• have a thread enabled julia build
• include the julia file
• set OPENBLAS_NUM_THREADS to 1
• set JULIA_NUM_THREADS to desired number of threads
• call the test_thread(dataset_path) method in the example script

[andreasnoack]

I think this is a duplicate of #15871 and #15276. Could you try running the threading benchmarks with commit dc6b0de

[abhijithch]

Thanks @andreasnoack, also this is segfaulting for nthreads() > 30.

[ranjanan]

Just adding versioninfo() for reference:

Julia Version 0.5.0-dev+1918
Commit 80b14f3 (2015-12-24 13:04 UTC)
Platform Info:
  System: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Sandybridge)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.3

[ViralBShah]

I think this is not a dup. @abhijithch Didn't you use an older commit on julia master? Could you point out which commit you used, and is it possible to bisect and find where things broke?

[ViralBShah]

Oops - ignore. I see @ranjanan did post the commit.

[ViralBShah]

Cc @tanmaykm

[amitmurthy]

How many cores?

[ranjanan]

30 cores with 2 hardware threads per core.

[kpamnany]

There appear to be 3 different things here.

1. Performance regression. This doesn't seem connected with threads; with 1 thread alone, test_thread() completes in 60.011244 seconds on commit dc6b0de and in 75.866557 seconds on master. I think @andreasnoack is right about this part.
2. Segfault. Occurs with 32 threads on commit dc6b0de, and with 16 threads on master. This is Multithreading segmentation fault #15875.
3. Poor scalability. On commit dc6b0de, performance with 1, 2, 4, 8, and 16 threads is 60.011244, 53.702526, 49.711778, 47.373577, and 47.155389 seconds respectively. On inspection, update_user() and update_item() take ~1.5 and ~1.1 seconds on 138493 users and 26744 items. This should show better benefit from threads. Barrier time is negligible which means the loops are regular and load is balanced. How big and unique is the data touched by each thread? Perhaps the cache is being thrashed?

[yuyichao]

I suspect the performance/scalability issues are probably all/mainly #15276. This includes #15740 #15871 and #15874

[kpamnany]

Threads still scale on master so I think the scalability issue here is unrelated to the performance regression.

[ViralBShah]

Given that each user will have different interactions with the items set, there shouldn't be much of cache thrashing. You can randomply permute the rows and columns of the R matrix and check if it changes performance. At each iteration, a submatrix is selected from R to work on.

[abhijithch]

On the commit dc6b0de segfault occured for 30 threads itself. The performance with 1, 2, 5, 10, 20 and 29 threads is 68, 61, 58, 53, 58 and 79 seconds respectively.

[kpamnany]

@ViralBShah: what I meant was, is it possible that each thread is bringing too much data into the cache and evicting other threads' data?

[pingzou]

I want to try multi-threading, but when I type nthreads() in the REPL, the
result is 1. Would you please tell me how to set JULIA_NUM_THREADS?

[nalimilan]

@pingzou Just start Julia with e.g. JULIA_NUM_THREADS=2 julia. (But note this only works with a recent git master.)

[ViralBShah]

@abhijithch Can you try this again and see where we are?

[KristofferC]

Closing due to inactivity.