Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

http_client hangs when upload many files at the same time #1245

Open
LiShuangcheng84 opened this issue Sep 19, 2019 · 3 comments
Open

http_client hangs when upload many files at the same time #1245

LiShuangcheng84 opened this issue Sep 19, 2019 · 3 comments

Comments

@LiShuangcheng84
Copy link

the code:
pplx::task RestAdapter::UploadTask(const std::string& url, const std::string& fid,
const std::string& filePath, RestCallback callback, void* context)
{
return file_stream<uint8_t>::open_istream(filePath).then([=](istream stream) {
http_client_config config;
config.set_timeout(utility::seconds(1));
http_client client("http://" + url);
return client.request(methods::PUT, uri_builder().append_path(fid).to_string(), stream)
.then([=] (http_response response) {
// stream.close();
if (response.status_code() != status_codes::Created) {
LOG_ERROR("Upload:%s to http://%s/%s failed:%u.", filePath.c_str(), url.c_str(), fid.c_str(), response.status_code());
callback(false, context);
} else {
LOG_DEBUG("Upload:%s to http://%s/%s successfully:%u.", filePath.c_str(), url.c_str(), fid.c_str(), response.status_code());
callback(true, context);
}
});
});
}

all asio threads have same bt like this:
(gdb) bt
#0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007fee17e994dc in std::condition_variable::wait(std::unique_lockstd::mutex&) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2 0x00007fee183b0bc5 in std::condition_variable::wait<pplx::details::event_impl::wait(unsigned int)::{lambda()#1}>(std::unique_lockstd::mutex&, pplx::detai ls::event_impl::wait(unsigned int)::{lambda()#1}) (this=0x7fee0803fb20, __lock=..., __p=...) at /usr/include/c++/4.8/condition_variable:93
#3 0x00007fee183ab942 in pplx::details::event_impl::wait (this=0x7fee0803faf8, timeout=4294967295)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplxlinux.h:98
#4 0x00007fee183aba09 in pplx::details::event_impl::wait (this=0x7fee0803faf8)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplxlinux.h:113
#5 0x00007fee183acfa8 in pplx::details::_TaskCollectionImpl::_Wait (this=0x7fee0803faf8)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplx.h:161
#6 0x00007fee183acf8e in pplx::details::_TaskCollectionImpl::_RunAndWait (this=0x7fee0803faf8)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplx.h:158
#7 0x00007fee183adaf7 in pplx::details::_Task_impl_base::_Wait (this=0x7fee0803fa98)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplxtasks.h:1777
#8 0x00007fee183b20c2 in pplx::task::wait (this=0x7fedfbffe380)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplxtasks.h:3601
#9 0x00007fee183aeeb2 in pplx::task::wait (this=0x7fedfbffe380)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/pplx/pplxtasks.h:4594
#10 0x00007fee183e720b in Concurrency::streams::details::basic_file_buffer::basic_file_buffer (this=0x7fedb0001520, __in_chrg=)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/cpprest/filestream.h:99
#11 0x00007fee183e729e in Concurrency::streams::details::basic_file_buffer::basic_file_buffer (this=0x7fedb0001520, __in_chrg=)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/cpprest/filestream.h:106
#12 0x00007fee183f497a in std::_Sp_counted_ptr<Concurrency::streams::details::basic_file_buffer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose (
this=0x7fedb0001640) at /usr/include/c++/4.8/bits/shared_ptr_base.h:290
#13 0x0000000000456784 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fedb0001640)
at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
#14 0x0000000000455ffd in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::
__shared_count (this=0x7fedb8001598, __in_chrg=)
at /usr/include/c++/4.8/bits/shared_ptr_base.h:546
#15 0x00007fee183b09d4 in std::__shared_ptr<Concurrency::streams::details::basic_streambuf, (__gnu_cxx::_Lock_policy)2>::
__shared_ptr (
this=0x7fedb8001590, __in_chrg=) at /usr/include/c++/4.8/bits/shared_ptr_base.h:781
#16 0x00007fee183b0a14 in std::shared_ptr<Concurrency::streams::details::basic_streambuf >::~shared_ptr (this=0x7fedb8001590,
__in_chrg=) at /usr/include/c++/4.8/bits/shared_ptr.h:93
#17 0x00007fee183b39c4 in Concurrency::streams::streambuf::~streambuf (this=0x7fedb8001588, __in_chrg=)
at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/cpprest/astreambuf.h:876
#18 0x00007fee183e4de6 in Concurrency::streams::details::basic_istream_helper::~basic_istream_helper (this=0x7fedb8001588,
__in_chrg=) at /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/include/cpprest/streams.h:53
#19 0x00007fee183df524 in __gnu_cxx::new_allocator<Concurrency::streams::details::basic_istream_helper >::destroy<Concurrency::streams::details: :basic_istream_helper > (this=0x7fedb8001580, __p=0x7fedb8001588) at /usr/include/c++/4.8/ext/new_allocator.h:124
#20 0x00007fee183daded in std::allocator_traits<std::allocator<Concurrency::streams::details::basic_istream_helper > >::_S_destroy<Concurrency:: streams::details::basic_istream_helper > (__a=..., __p=0x7fedb8001588) at /usr/include/c++/4.8/bits/alloc_traits.h:281
#21 0x00007fee183d71a7 in std::allocator_traits<std::allocator<Concurrency::streams::details::basic_istream_helper > >::destroy<Concurrency::str eams::details::basic_istream_helper > (__a=..., __p=0x7fedb8001588) at /usr/include/c++/4.8/bits/alloc_traits.h:405
#22 0x00007fee183d08f5 in std::_Sp_counted_ptr_inplace<Concurrency::streams::details::basic_istream_helper, std::allocator<Concurrency::streams: :details::basic_istream_helper >, (__gnu_cxx::_Lock_policy)2>::_M_dispose (this=0x7fedb8001570)
at /usr/include/c++/4.8/bits/shared_ptr_base.h:407
#23 0x0000000000456784 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x7fedb8001570)
at /usr/include/c++/4.8/bits/shared_ptr_base.h:144
---Type to continue, or q to quit---
#24 0x00007fee167f5c01 in web::http::details::http_msg_base::set_body(Concurrency::streams::basic_istream const&, std::string const&) ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#25 0x00007fee167e12ca in web::http::client::details::request_context::complete_headers() ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#26 0x00007fee16906144 in web::http::client::details::asio_context::read_headers() ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#27 0x00007fee1690b747 in web::http::client::details::asio_context::handle_status_line(boost::system::error_code const&) ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#28 0x00007fee168eec93 in boost::asio::detail::read_until_delim_string_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_ser viceboost::asio::ip::tcp >, std::allocator, boost::_bi::bind_t<void, boost::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::er ror_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptrweb::http::client::details::asio_context >, boost::arg<1> ()()> > >::operator()(boost:: system::error_code const&, unsigned long, int) () from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#29 0x00007fee168ef2db in boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::read_until_delim_string_op<boost::a sio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_serviceboost::asio::ip::tcp >, std::allocator, boost::_bi::bind_t<void, boost ::_mfi::mf1<void, web::http::client::details::asio_context, boost::system::error_code const&>, boost::_bi::list2<boost::_bi::value<std::shared_ptr<web::http::c lient::details::asio_context> >, boost::arg<1> (
)()> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation* , boost::system::error_code const&, unsigned long) () from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#30 0x00007fee1685f69f in boost::asio::detail::task_io_service::run(boost::system::error_code&) ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#31 0x00007fee168e0c17 in boost::asio::detail::posix_thread::func<(anonymous namespace)::threadpool_impl::add_thread()::{lambda()#1}>::run() ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#32 0x00007fee1685022f in boost_asio_detail_posix_thread_function ()
from /workspace/code/Venom/code/storage.client/../../third_party/cpprestsdk/libs/libcpprest.so.2.10
#33 0x00007fee186b9184 in start_thread (arg=0x7fedfbfff700) at pthread_create.c:312
#34 0x00007fee1790a03d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

@LiShuangcheng84
Copy link
Author

image

@zzzhr1990
Copy link

zzzhr1990 commented Dec 8, 2019

Seems the max concurrent task limit is 64 (the default thread pool size?)
(Just on OSX/Linux, On windows, everything is ok)

int main() {
    const int MAX_THREAD_ID = 62; // if change this above 63, program will be blocked forever.
    std::vector<pplx::task<int>> u;
    web::http::client::http_client client(_XPLATSTR("https://www.microsoft.com"));
    for(int i =0;i<= MAX_THREAD_ID;i++)
    {
        auto t = pplx::create_task([&client]{
            web::http::http_request req;
            req.set_method(web::http::methods::GET);
            try {
                return static_cast<int>(client.request(req).get().status_code());
            }catch (const std::exception& exx) {
                std::cout << exx.what() << std::endl;
            }
            return -1;
        });
        u.emplace_back(t);
    }

    auto complete = pplx::when_all(u.begin(), u.end()).get();
    for(auto re: complete){
        std::cout << "result: " << re << std::endl;
    }

    std::cout << "-fin-" << std::endl; // 200

    std::cin.get();
    return 0;
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants
@LiShuangcheng84 @zzzhr1990 and others