CRM-20926 - Allow extensions to flag PHPIDS html/json fields

[totten]

Background

PHPIDS scans all HTTP inputs and applies heuristics to identify malicious
submissions. It relies on a having configuration about the list of fields
we'll process.

NOTE: This PR includes/depends on #10708.

Before

The function createConfigFile() produces the standard configuration as an
INI file. The configuration was represented as an unalterable string.

After

• The configuration is represented as an array.
• The configuration is generated via createStandardConfig()
• The configuration file is no longer used.

Acceptance Criteria

• PHPIDS still protects most page-requests.
• The pre-existing PHPIDS policies still apply.
• Individual inputs for individual pages – such as the civicase_reload involved with CRM-20924 – can be flagged as html, json, or other exceptions to the PHPIDS heuristics.

[totten]

jenkins, test this please

[seancolsen]

"Allow extensions to..." — This caught my eye. Would it be possible to add some documentation to the Developer Guide to go along with this PR so that extensions developers can learn how to do whatever it is that this PR would allow them to do?

[totten]

@seanmadsen This should be ready for you to give a whirl. I've posted some docs in civicrm/civicrm-dev-docs#227 along with an example of triggering the IDS in https://gist.github.com/totten/27cc1bde206fe17371c8a6d7cb361e49

[totten]

@seanmadsen Updates following our discussion

• Rebased PR and resolved conflicts
• Fixed the issue where "exception" and "exceptions" get confused. Updated the docs PR accordingly
• Fixed the undefined index
• Improved test coverage
• Tested which policies (exceptions, html, json) accept the example data from https://gist.github.com/totten/27cc1bde206fe17371c8a6d7cb361e49 -- these are the results:
  • This allows the scary data: $items['civicrm/admin']['ids_arguments']['exceptions'][] = 'foo';
  • This rejects the scary data: $items['civicrm/admin']['ids_arguments']['html'][] = 'foo';
  • This rejects the scary data: $items['civicrm/admin']['ids_arguments']['json'][] = 'foo';

(The commit#'s have all changed following rebase, but the only new thing is 615e575.)

[seancolsen]

I've reviewed this PR by looking over the code and running it locally.

• ✅ It's a worthy pursuit
• ✅ It's documented
• ✅ It includes tests for new functionality
• ✅ Overall, it behaves as advertised
• ✅ Acceptance criteria is met
  • ✅ PHPIDS still protects most page-requests.
  • ✅ The pre-existing PHPIDS policies still apply.
  • ✅ Individual inputs for individual pages can be flagged as html, json, or other exceptions to the PHPIDS heuristics.

Good to go!

[totten]

Thank you @seanmadsen!

It was really helpful how your review involved trying it out -- that step found the exception/exceptions bug which (a) wasn't visible in my original test scenario and (b) would have been very hard to spot from just reading the code. Nice.

I'll rebase to resolve the recent conflict and move the upgrade logic to match the new version#. Then it should be merge-on-pass.

[seamuslee001]

@totten tests have passed