Incentivizing Node API operators (premium endpoint service for clients)

[manikey123]

User Story
Incentivizing Node API operators based on the desired user bandwidth for premium services. The APIS are limited as a protection from DDOS. Configuring limits dynamically based on user preference from the node operator side to generate revenue for the Node API operators. As the functionality affects core and other client teams, discussion/ suggestions/ feedback on the below approaches would be needed.

Suggested three approaches:

Approach 1)

1. Bitshares Node API operators have a feature from core- Change hard-coded limitations in API's to configurable #782 , Review limitations on API's #783 (completed) and Make more API limitations configurable #1733 (ongoing) to configure the API limits and support the user required bandwidth.
2. Creating API for getting the configured values based on every node.(the client will be aware of the configured values based on the nodes its connected to)

Disadvantage:

1. The maximum possible threshold is to be determined based on the latency of the node to avoid DDOS.
2. The higher threshold limit can also be hard coded for the desired APIs by the Node API operators.

Approach 2)

1. Using the Elastic search plugin.
2. Map the existing APIs and functionality to the ES APIs.
3. For the get_account_history map the incentive functionality to depth of the query.

Disadvantage:

1. Not all the APIs are mapped. Better filters needs to be identified and set.
2. Account history plugin account cannot be used if the ES plugin is used.
3. Extra space will be needed aproximately 500 GB. (Current resync of Elastic Search data is 392GB + witness_node data of 71GB. @ryanRfox )
   https://github.com/bitshares/bitshares-core/wiki/ElasticSearch-Plugin

Approach 3) Combines the best of above two approaches and overcomes disadvantages. (Hybrid approach)

1. Let the client choose and toggle between the ES nodes and the normal Node APIs for the best experience
2. The configurations for the same can be done between the Node APIs and the ES node as per the need
3. This is already achieved in the BTS UI wallet where in the users can choose their Node APIs.
4. The ES nodes can also be selected as a hybrid approach.
5. @sschiessl-bcp , @startailcoon @oxarbitrage can you plz share your inputs in the above context on the current integration of the ES node in the UI wallet.
   (Full history will fail if total OPs is beyond 10,000 bitshares-ui#2997 )
   (Implement a 'get_historic_feed_price' call bitshares-explorer-api#53)
   The below functionality would be common to all the 3 approaches based on the incentivizing feature. Preference is starting out with Approach 1 due to ease of implementation in Core.
   Approach 2 and approach 3 needs discussion.

Common parts for all three above approaches:

1. Create a whitelist at the Node-API end of the accounts which need special configurations @xeroc . Can you please share your inputs from:
   ([WIP] Feature/api login - requesting feedback #1880)
2. For a whitelisted account with special configurations, reconfigure the API limits in memory with the given value.(Discussion on the limits is needed to avoid DDOS)
3. To add the accounts to the whitelist, Node API operators can take a fee from the clients. Node API operators can use the direct debit feature to debit the user accounts in a recurring manner.(Not sure if this should be handled by bitshares core or the Node API operators)
4. Else proceed with default behavior
5. The UI or the client applications can know the threshold values by using a get request. (get_configured_api_limits) which would give the API limits configured by the desired selected node based on the whitelist. ([2][manikey123] Ensure back end API changes reflected to front end based on config file updates bitshares-ui#2406)

Extended features:
a need for a 'stats' page to be built displaying all the stats from the paid and unpaid node operators so it is clear what you are paying for. That is relevant to Dexbot users, UI, mobile and the wider community too. (Idea from George Harrap)

Impacts
Describe which portion(s) of BitShares Core may be impacted by your request. Please tick at least one box.

• API (the application programming interface)
• Build (the build process or something prior to compiled code)
• CLI (the command line wallet)
• Deployment (the deployment process after building such as Docker, Travis, etc.)
• DEX (the Decentralized EXchange, market engine, etc.)
• P2P (the peer-to-peer network for transaction/block propagation)
• Performance (system or user efficiency, etc.)
• Protocol (the blockchain logic, consensus, validation, etc.)
• Security (the security of system or user data, etc.)
• UX (the User Experience)
• Other (please add below)

Additional Context (optional)
Add any other context about your request here.

CORE TEAM TASK LIST

• Evaluate / Prioritize Feature Request
• Refine User Stories / Requirements
• Define Test Cases
• Design / Develop Solution
• Perform QA/Testing
• Update Documentation

[abitmore]

About API mapping, there was a project: https://github.com/bitshares/btsproxy, rather than changing core code, I think that infrastructure is clearer and easier to extend (we can get help from developers who don't write C++).

For authentication, currently we only support api-access file for easy authentication. To update it, we have to restart the node. Even if we can reload on the fly, since the file is not well structured for frequent read, it would be inefficient to reload when the file gets large, also it costs RAM if we keep the file data in memory. I think it would be good if we can integrate the authentication process with another professional software which provides an authentication service (E.G. support SASL).

It's not the best idea to use active key for the process, due to potential risks on clients. Currently, perhaps memo key is better. With #1847, users can add other data to the chain, so can define a key there. If decided to use signed transaction for authentication, please also consider BSIP40, with it users can define special keys for special operations.

The designed workflow is not really compatible with workflow of current clients, specifically, the use of active key. For example, in reference UI, a user usually connect to a server, query for some information first, will only login when need to broadcast a transaction. It would be too early for the user to use the active key. It would be good if the user can choose when to authenticate.

Later, we could define limitations e.g. number of requests per seconds, bandwidth limit and etc for each user based on authentication.

I think it's better to write an informational BSIP for the whole concept, especially with clear specifications.

About implementation, the PR added a login_signed API. Alternatively, due to the optional parameter support (bitshares/bitshares-fc#126), we can extend current login API, keep the status that currently the login API is the entry point of the authentication process. For example, extend login API from login(user,pass_string) to login(user,auth_string,optional_auth_method). There are reasons for this. For example, currently we support Basic HTTP Authentication for clients, it's best if the new authentication method has similar support.