★ wanayoo — archive 1999 https://github.com/taskflow/taskflow/issues/158Nouvelle recherche | Portail wanayoo
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Executor::this_worker_id() not working in larger programs #158

Open
tomager opened this issue Apr 14, 2020 · 9 comments
Open

Executor::this_worker_id() not working in larger programs #158

tomager opened this issue Apr 14, 2020 · 9 comments
Labels

Comments

@tomager
Copy link

@tomager tomager commented Apr 14, 2020 •

Some logging at tf::Executor::_spawn:
spawn 0 _per_thread()=0x7f56323bf6f8 (thread 140008186705664)
spawn 1 _per_thread()=0x7f5631b6e6f8 (thread 140008177985280)

Now, within a task run from above Executor, here is logging from within this_worker_id:
this_worker_id _per_thread()=0x7f5631b6e658 (thread 140008177985280)

Which results in:
exec.this_worker_id() = -1

The thread number is the result of std::this_thread::get_id(). Note the varying &_per_thread() value above for the same thread 140008177985280.

What appears to be happening is that because _per_thread is inlined, it is duplicated multiple times in the program, and hence there are multiple instantiated PerThread structs per actual thread (one per instantiated _per_thread in the final link). Only the one touched by Executor::_spawn is initialized with the worker info.

Moving Executor::_per_thread() into a single translation unit fixes the problem. I would submit a patch for that, but it would render cpp-taskflow no longer a header-only library, so I am creating this issue instead. I am happy to generate a patch with guidance about preferred solution.

@tsung-wei-huang
Copy link
Member

@tsung-wei-huang tsung-wei-huang commented Apr 14, 2020

I am a bit confused. this_worker_id only returns a feasible number [0, N) if the caller thread is a worker of the executor. If the caller thread is not a worker of the executor (spawned from the executor), it returns -1. Are you calling this_worker_id from a worker thread or a free-standing thread?

@tomager
Copy link
Author

@tomager tomager commented Apr 14, 2020

From a worker thread, as the logging I posted indicates.

@tomager
Copy link
Author

@tomager tomager commented Apr 14, 2020 •

There is a detail here about shared objects that makes this less obvious. Apparently gcc aggregates thread_local setup to program / shared library boundaries. The two interacting translation units have to be across that boundary. Here's a test case that reproduces the problem:

file1.cc:

#include <iostream>
#include <taskflow/taskflow.hpp>

void call_file2(tf::Executor& exec);

int main(int argc, const char** argv) {

  tf::Executor exec(2);
  tf::Taskflow tf;
  tf.emplace([&exec](){
      std::cout<<"from file1: exec.this_worker_id() = "<<exec.this_worker_id()<<std::endl;
      call_file2(exec);
    });
  exec.run(tf).wait();

  return 0;
}

file2.cc:

#include <taskflow/taskflow.hpp>

void call_file2(tf::Executor& exec) {
  std::cout<<"from file2: exec.this_worker_id() = "<<exec.this_worker_id()<<std::endl;
}

Build like this:
g++ -fPIC -O3 -I cpp-taskflow file1.cc -c -o file1.o && g++ -fPIC -O3 -I cpp-taskflow file2.cc -shared -o file2.so && g++ -fPIC file1.o file2.so -lpthread -o out && LD_LIBRARY_PATH=`pwd` ./out

Output is:
from file1: exec.this_worker_id() = 1
from file2: exec.this_worker_id() = -1

g++ (Ubuntu 7.2.0-8ubuntu3.2) 7.2.0

@tsung-wei-huang
Copy link
Member

@tsung-wei-huang tsung-wei-huang commented Apr 15, 2020

@tomager Thanks for bringing this up! I wonder if there is any way to solve this without making taskflow non-header only. I recall in C++17 inline variable may be a good way to do. What is your take?

@tomager
Copy link
Author

@tomager tomager commented Apr 17, 2020

So I can get file1/file2 to see the same static inline variable, but any use of thread_local doesn't want to play nice and uses the old non-inline static rules. Direct use of "static inline thread_local" doesn't compile (on gcc 7.2 anyway).

The only workable solution I see is to use a static inline struct to initialize some thread local data with pthread/win32 apis rather than thread_local, like

  struct PerThread0 {
    std::mutex lock;
    std::unique_ptr<PerThread1> pt1;
  };
  inline static PerThread0 pt0;
  void _init_per_thread() { // call from Executor ctor
    std::unique_lock<std::mutex> g(pt0.lock);
    if (!pt0.pt1)
      pt0.pt1.reset(new PerThread1); // manually init tls here
  }

That's a little bit of fuss but not too wacky, and keeps the library header-only. First step should be to try "static inline thread_local PerThread pt" on a more recent compiler and see if that works as is.

@tomager
Copy link
Author

@tomager tomager commented Apr 17, 2020

More specifically, the patch below fixes the problem and keeps the library header only, using only the non-thread_local "static inline pt0". Handling win32 isn't much different I believe (ifdef on WIN32, use TlsAlloc, TlsGetValue, TlsSetValue). Thoughts?

diff --git a/taskflow/core/executor.hpp b/taskflow/core/executor.hpp
index 8c15e50..1187b3d 100644
--- a/taskflow/core/executor.hpp
+++ b/taskflow/core/executor.hpp
@@ -14,6 +14,8 @@
 #include <numeric>
 #include <cassert>

+#include <pthread.h>
+
 #include "tsq.hpp"
 #include "notifier.hpp"
 #include "observer.hpp"
@@ -30,7 +32,7 @@ an efficient work-stealing scheduling algorithm to run a taskflow.

 */
 class Executor {
-
+
   struct Worker {
     size_t id;
     size_t victim;
@@ -46,6 +48,7 @@ class Executor {
     Worker* worker {nullptr};
   };

+
 #ifdef TF_ENABLE_CUDA
   struct cudaDevice {
     std::vector<cudaStream_t> streams;
@@ -74,6 +77,29 @@ class Executor {
     */
     ~Executor();

+  struct PerThread1 {
+    PerThread1() {
+      pthread_key_create(&key, NULL);
+    }
+    ~PerThread1() {
+      pthread_key_delete(key);
+    }
+    PerThread1(const PerThread1& rhs) = delete;
+    PerThread1& operator= (const PerThread1& rhs) = delete;
+    pthread_key_t key;
+  };
+  struct PerThread0 {
+    std::mutex lock;
+    std::unique_ptr<PerThread1> pt1;
+  };
+  inline static PerThread0 pt0;
+
+  void _init_per_thread() {
+    std::unique_lock<std::mutex> g(pt0.lock);
+    if (!pt0.pt1)
+      pt0.pt1.reset(new PerThread1);
+  }
+
     /**
     @brief runs the taskflow once

@@ -225,7 +251,7 @@ class Executor {

     std::unique_ptr<ExecutorObserverInterface> _observer;

-    PerThread& _per_thread() const;
+  //  PerThread& _per_thread() const;

     bool _wait_for_task(Worker&, Node*&);

@@ -269,6 +295,8 @@ inline Executor::Executor(size_t N, size_t M) :
     TF_THROW("no gpu workers to execute cudaflows");
   }

+  _init_per_thread();
+
   for(int i=0; i<NUM_DOMAINS; ++i) {
     _num_actives[i].store(0, std::memory_order_relaxed);
     _num_thieves[i].store(0, std::memory_order_relaxed);
@@ -299,6 +327,8 @@ inline Executor::Executor(size_t N) :
   if(N == 0) {
     TF_THROW("no cpu workers to execute taskflows");
   }
+
+  _init_per_thread();

   for(int i=0; i<NUM_DOMAINS; ++i) {
     _num_actives[i].store(0, std::memory_order_relaxed);
@@ -351,16 +381,10 @@ inline size_t Executor::num_topologies() const {
   return _num_topologies;
 }

-// Function: _per_thread
-inline Executor::PerThread& Executor::_per_thread() const {
-  thread_local PerThread pt;
-  return pt;
-}
-
 // Function: this_worker_id
 inline int Executor::this_worker_id() const {
-  auto worker = _per_thread().worker;
-  return worker ? static_cast<int>(worker->id) : -1;
+  auto worker = (Worker*)pthread_getspecific(pt0.pt1->key);
+  return worker ? static_cast<int>(worker->id) : -1;
 }

 // Procedure: _spawn
@@ -380,8 +404,7 @@ inline void Executor::_spawn(size_t N, Domain d) {

     _threads.emplace_back([this] (Worker& w) -> void {

-      PerThread& pt = _per_thread();
-      pt.worker = &w;
+       pthread_setspecific(pt0.pt1->key, &w);

       Node* t = nullptr;

@@ -642,7 +665,7 @@ inline void Executor::_schedule(Node* node, bool bypass_hint) {
   const auto d = node->domain();

   // caller is a worker to this pool
-  auto worker = _per_thread().worker;
+  auto worker = (Worker*)pthread_getspecific(pt0.pt1->key);

   if(worker != nullptr && worker->executor == this) {
     if(bypass_hint) {
@@ -685,7 +708,7 @@ inline void Executor::_schedule(PassiveVector<Node*>& nodes) {
   }

   // worker thread
-  auto worker = _per_thread().worker;
+  auto worker = (Worker*)pthread_getspecific(pt0.pt1->key);

   // task counts
   size_t tcount[NUM_DOMAINS] = {0};
@tomager
Copy link
Author

@tomager tomager commented Apr 17, 2020

Building with gcc head (g++ (GCC) 10.0.1 20200416 (experimental)) the following simpler patch works with naive use of "static inline thread_local PerThread pt". That doesn't have pthread/win32 dependency, but it will break the library for recent versions of gcc, and maybe all versions of clang/msvc. Those would need testing anyway. The manual tls method I posted earlier seems like the safer way to go, at least for the near future. Let me know your thoughts on how to proceed.

diff --git a/taskflow/core/executor.hpp b/taskflow/core/executor.hpp
index 8c15e50..c9a0b93 100644
--- a/taskflow/core/executor.hpp
+++ b/taskflow/core/executor.hpp
@@ -30,7 +30,7 @@ an efficient work-stealing scheduling algorithm to run a taskflow.

 */
 class Executor {
-
+
   struct Worker {
     size_t id;
     size_t victim;
@@ -43,9 +43,16 @@ class Executor {
   };

   struct PerThread {
-    Worker* worker {nullptr};
+    Worker* worker;
+    // ctor required to initialize worker, maybe related to
+    // https://stackoverflow.com/questions/53408962/try-to-understand-compiler-error-message-default-member-initializer-required-be
+    PerThread()
+      : worker(nullptr) {
+    }
   };
+  static inline thread_local PerThread pt;

+
 #ifdef TF_ENABLE_CUDA
   struct cudaDevice {
     std::vector<cudaStream_t> streams;
@@ -224,8 +231,6 @@ class Executor {
     std::atomic<bool>   _done {0};

     std::unique_ptr<ExecutorObserverInterface> _observer;
-
-    PerThread& _per_thread() const;

     bool _wait_for_task(Worker&, Node*&);

@@ -268,7 +273,7 @@ inline Executor::Executor(size_t N, size_t M) :
   if(M == 0) {
     TF_THROW("no gpu workers to execute cudaflows");
   }
-
+
   for(int i=0; i<NUM_DOMAINS; ++i) {
     _num_actives[i].store(0, std::memory_order_relaxed);
     _num_thieves[i].store(0, std::memory_order_relaxed);
@@ -351,16 +356,10 @@ inline size_t Executor::num_topologies() const {
   return _num_topologies;
 }

-// Function: _per_thread
-inline Executor::PerThread& Executor::_per_thread() const {
-  thread_local PerThread pt;
-  return pt;
-}
-
 // Function: this_worker_id
 inline int Executor::this_worker_id() const {
-  auto worker = _per_thread().worker;
-  return worker ? static_cast<int>(worker->id) : -1;
+  auto worker = pt.worker;
+  return worker ? static_cast<int>(worker->id) : -1;
 }

 // Procedure: _spawn
@@ -380,7 +379,6 @@ inline void Executor::_spawn(size_t N, Domain d) {

     _threads.emplace_back([this] (Worker& w) -> void {

-      PerThread& pt = _per_thread();
       pt.worker = &w;

       Node* t = nullptr;
@@ -642,7 +640,7 @@ inline void Executor::_schedule(Node* node, bool bypass_hint) {
   const auto d = node->domain();

   // caller is a worker to this pool
-  auto worker = _per_thread().worker;
+  auto worker = pt.worker;

   if(worker != nullptr && worker->executor == this) {
     if(bypass_hint) {
@@ -685,7 +683,7 @@ inline void Executor::_schedule(PassiveVector<Node*>& nodes) {
   }

   // worker thread
-  auto worker = _per_thread().worker;
+  auto worker = pt.worker;

   // task counts
   size_t tcount[NUM_DOMAINS] = {0};
@tsung-wei-huang
Copy link
Member

@tsung-wei-huang tsung-wei-huang commented Apr 20, 2020 •

I definitely prefer the last solution of using inline variable, but as you said, it's supported only by C++17. The problem is NCVV is not yet there. I will keep this an open issue and see what I can do in the future.

Perhaps the best solution for now is to resort the worker_id mapping to the client.

@tomager
Copy link
Author

@tomager tomager commented Apr 20, 2020

If you're willing to have the client deal with worker_id, why not just make the -1 case an error (eg, throw std::runtime_error). If a user encounters that (apparently not common), leave a note in the error or code telling them how to use the library as non-header-only. Like, adding -DTASKFLOW_NOT_HEADER_ONLY to their build and including a small static library built from cpp-taskflow CMakeLists.txt that contains the one definition of _per_thread. The header-definition of _per_thread would be wrapped by #ifndef TASKFLOW_NOT_HEADER_ONLY ... #endif etc.

I feel like that would be the most friendly for users today, and it's a trivial patch. In the future go with the c++17 method.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
2 participants
You can’t perform that action at this time.