3.2.0 witness_node hang, too many open files

[abitmore]

Bug Description
witness_node stops responding after started for a while but is running at high cpu %.
The log messages below are probably related.

2019-07-18T09:57:27 p2p:p2p_network_connect_loop p2p_network_connect_ ] 0: 0    node.cpp:470
2019-07-18T09:57:27 p2p:p2p_network_connect_loop   destroy_connection ] in destroy_connection() for    message_oriented_connection.cpp:285
2019-07-18T09:57:27 p2p:p2p_network_connect_loop   destroy_connection ] in destroy_connection() for    message_oriented_connection.cpp:285
2019-07-18T09:57:27 p2p:p2p_network_connect_loop p2p_network_connect_ ] 13 N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_6system12system_errorEEEEE: open: Too many open files
rethrow open: Too many open files: 
    {"0":0,"what":"open: Too many open files"}
    p2p  node.cpp:470 p2p_network_connect_loop   node.cpp:470

Impacts
Describe which portion(s) of BitShares Core may be impacted by this bug. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

Steps To Reproduce
Steps to reproduce the behavior (example outlined below):

1. Execute API call '...'
2. Using JSON payload '...'
3. Received response '...'
4. See error in screenshot

Expected Behavior
A clear and concise description of what you expected to happen.

Screenshots (optional)
If applicable, add screenshots to help explain process flow and behavior.

Host Environment
Please provide details about the host environment. Much of this information can be found running: witness_node --version.

• Host OS: [e.g. Ubuntu 18.04 LTS]
• Host Physical RAM [e.g. 4GB]
• BitShares Version: [e.g. 2.0.180425]
• OpenSSL Version: [e.g. 1.1.0g]
• Boost Version: [e.g. 1.65.1]

Additional Context (optional)
Add any other context about the problem here.

CORE TEAM TASK LIST

• Evaluate / Prioritize Bug Report
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[abitmore]

The file descriptor aren't closed properly after a websocket connection is closed. #1854 should fix it.

Steps to reproduce:

1. start a witness_node which acceps RPC connections

./witness_node --rpc-endpoint=127.0.0.1:8090

2. in another terminal, run lsof command to check the node's open files

lsof -np [pid_of_witness_node]

3. send an API request to witness_node

curl http://127.0.0.1:8090/

4. runlsof again, there would be one sock file descriptor left opened

witness_n 1981 ubuntu *770u     sock       0,8         0t0 260881922 protocol: TCP

5. repeat 3 and 4, there would be more files left opened.

[abitmore]

Fixed by #1854.