Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

REF dev/core#2571 Move reCAPTCHA code to extension #20588

Merged
merged 1 commit into from
Jul 10, 2022

Conversation

mattwire
Copy link
Contributor

@mattwire mattwire commented Jun 13, 2021

Overview

This moves reCAPTCHA code to extension for all core forms:

  • CRM_Contribute_Form_Contribution_Main - Recaptcha working as expected, both by using forced Recaptcha and using a profile with Recaptcha (for anonymous users).
  • CRM_Campaign_Form_Petition_Signature - Recaptcha working as expected (using profile) (For anonymous users).
  • CRM_Event_Form_Registration_Register - Getting a Recaptcha using a profile as expected.
  • CRM_Mailing_Form_Subscribe - I am getting a recaptcha as an anonymous user after going to ‘https://example.org/civicrm/mailing/subscribe' .
  • CRM_PCP_Form_PCPAccount - Since PCP just redirects to Contribution buttons after clicking on donate now, then there is a recaptcha on Contribution page.
  • CRM_Profile_Form_Edit - If one of the profiles has recaptcha enabled then it should be displayed.
  • CRM_Event_Form_Registration_Register - I am getting a recaptcha only for primary participant (using Profile), For additional participant if I decide to skip, its working fine.

Before

reCAPTCHA loaded via core form code.

After

reCAPTCHA loaded via extension code.

Technical Details

tpl files are still included via core templates at the moment.

Comments

@totten @monishdeb @eileenmcnaughton

@civibot
Copy link

civibot bot commented Jun 13, 2021

(Standard links)

@civibot civibot bot added the master label Jun 13, 2021
public static function checkAndAddCaptchaToForm($formName, &$form) {
$addCaptcha = FALSE;
$ufGroupIDs = [];

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mattwire I think there are a few common patterns in this function definition, handled under each switch-cases, which can be avoided by:

  1. Removing all if (!CRM_Core_Session::getLoggedInContactID()) check by:
  public static function checkAndAddCaptchaToForm($formName, &$form) {
    if (CRM_Core_Session::getLoggedInContactID()) {
       return;
    }
...
  1. Getting the $ufGroupIDs assignment in one place :
  public static function checkAndAddCaptchaToForm($formName, &$form) {
      if (CRM_Core_Session::getLoggedInContactID()) {
         return;
      }
     $ufGroupIDs  = method_exists($form, 'getUFGroupIDs') ? $form->getUFGroupIDs() : [];
     ...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@monishdeb I agree with those changes but that would change the existing behaviour which I'm trying to avoid for initial move of the code. As you can see there are inconsistencies - not all forms check if the contact is logged in for example when they probably should.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm agree.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually @monishdeb you were right - I've moved that to a "standard conditions" check.

@mattwire mattwire force-pushed the recaptcharefactor branch 3 times, most recently from ed86a81 to e0098b9 Compare July 13, 2021 15:13
@homotechsual
Copy link
Contributor

homotechsual commented Jul 28, 2021

(CiviCRM Review Template WORD-1.2)

@eileenmcnaughton
Copy link
Contributor

eileenmcnaughton commented Jul 29, 2021

Other than tests the thing I worry about is interfaces and how stable they are if we are implying them. There are 2 interface-like aspects here

  1. how we recommend to add captcha the form
  2. the way the captcha code gets the uf ids.

In the case of 2 the main concern is to protect us from would be copy & pasters & adding a comment warning in this section would address that.

For the first however - I think we are setting up something people are going to expect to never change so I think we should decide what we want the interface to be.

Potentially we should add a function to CRM_Core_Form that would look something like

public function addCaptchaToForm($ufGroupIDs) {
  // for the factory bit the right syntax is 'whatever Tim says it is'
  // The geocoder equivalent looks like  $provider = CRM_Utils_GeocodeProvider::getConfiguredProvider();
  $capcha  = CRM_Core_Config()->getCapchaService();
  $capcha->addToForm($this, $ufGroupIDs);
}

If we documented the agreed variant of that & added tests I think we could commit to keeping that working & not changing the signature at any future point

@seamuslee001 @colemanw @demeritcowboy @totten

@mattwire if you want to put the readme changes in a separate PR & add warnings to copy&pasters around using that function from extension code I think the bulk of this is mergeable if it doesn't imply any new supported interfaces (although obviously .... it really should have tests) - a big thanks to @homotechsual for testing

@mattwire
Copy link
Contributor Author

mattwire commented Aug 1, 2021

@eileenmcnaughton At this point we are not talking about making this a standalone extension although that is the end goal. This is really still just refactoring to extract into a core extension which generally doesn't require the addition of tests and definition of external interfaces. I completely agree that they would be essential before making this a "standalone extension". The recaptcha extension is currently marked as "hidden" and, in my opinion, fully compliant with the original intent/contract for core extensions in that they would allow for gradually moving functionality to an extension without having to tackle some of the tougher, architectural issues such as interfaces.

Regarding:

  1. how we recommend to add captcha the form

None of this work is changing that interface. I agree that we do need to define a supported interface before this becomes unhidden/standalone but feel that is out of scope of this part of the work. Also note that the end-goal here is to have "no reference" to recaptcha in civicrm core so I don't think adding a method to CRM_Core_Form is the right approach.

Example: https://github.com/veda-consulting-company/uk.co.vedaconsulting.gdpr/blob/master/CRM/Gdpr/Form/UpdatePreference.php#L98

  1. the way the captcha code gets the uf ids.

I think this is a bit of a side-issue. Currently a lot of the control over displaying reCAPTCHA is defined by a flag in the profile but I'm not even sure this will be required in the future. Generally the display of reCAPTCHA seems to be based on "are you logged in?" and a generic setting for all public/anonymous forms is more likely in the future - especially for "hidden" form protection options such as honeypot / invisible reCAPTCHA etc.
For example, we might even be able to reach agreement to completely remove the configuration in profiles before this becomes a standalone extension.

@eileenmcnaughton
Copy link
Contributor

@mattwire I would disagree with you that " just refactoring to extract into a core extension" "generally doesn't require the addition of tests" - I have added tests to all the code I have refactored into core extensions. I feel a bit gaslit when you say that to be honest.

Regarding whether defining an external interface is required - the trigger point for needing to define that in this case, as I mentioned was you writng a readme telling people how to add the code to their extensions - at that point you are defining a new interface. That function name could change at any time but by documenting it you are implying that it won't and that extensions can call it from outside of core - that is what an interface is.

@mattwire mattwire force-pushed the recaptcharefactor branch from e0098b9 to c4adfa4 Compare August 2, 2021 22:11
@mattwire
Copy link
Contributor Author

mattwire commented Aug 2, 2021

@eileenmcnaughton I see what you mean about the readme - I'd actually forgotton I'd written that level of documentation. Now updated to indicate that there is no supported method.
The more I think about this the more I think it's likely that an alternative "form protection" extension would replace rather than build on this extension because in most cases the logic could be much simpler - eg. "add to all anonymous forms" and indeed many of these (including invisible ReCAPTCHA) provide their own logic as to when to add / when to display so CiviCRM doesn't need to handle it.
That would mean the scope here really is just "move everything related to ReCAPTCHA to the core extension" so CiviCRM core can be shipped without reference to Google/ReCAPTCHA - which also fits with the requirement of the funder in this case which is to remove references to third-party services provided by corporations.

Regarding tests, generally agree they're a good idea and you've done a good job. But looking at what we have as core extensions currently the "logic" ones such as sequentialcreditnotes and contributioncancelactions have a good set of tests. But the "UI" ones don't have tests - such as ckeditor4 and greenwich theme. I feel like ReCAPTCHA fits in the "UI" category and I don't see an easy way to test (indeed the logic of ReCAPTCHA is to prevent automated submission/testing) - the 7 forms which it is added to in core can be easily tested manually but validating that it is added using a test doesn't seem so straightforward.

@mattwire
Copy link
Contributor Author

mattwire commented Aug 2, 2021

@eileenmcnaughton I meant to say I've updated the README now to be clearer.

@eileenmcnaughton
Copy link
Contributor

@mattwire I would personally have added tests to at least one form at either the point of extracting getUFIds() (the previous PR) or at the point when the code started using those functions (this pr) as those are new code & new logic for inserting the captchas - not just moved code and it's not using otherwise-tested interfaces like the ckeditor does. However, I'll bow out & someone else might be comfortable merging this without adding tests.

Regarding the readme update - I think it would be better to fully remove any 'how to' for interacting with the extension in an unsupported way from the PR template & the readme. The comment block for the function should have some suitably scary messages & internal tags added.

On the scope and goal of the work - I can see that is still very much an open question on the gitlab issue so I won't get into it here - but it looks like it does need to be agreed in gitlab before more PRs

@mattwire
Copy link
Contributor Author

@eileenmcnaughton I've updated readme/comments/PR as requested. Think this should be ok now?

@jaapjansma
Copy link
Contributor

Betty and I are reviewing PR's and we came across this one. We have asked @eileenmcnaughton in the chat for a comment on this PR as it looks like this PR is stalled.

@jaapjansma
Copy link
Contributor

I have spoken with @eileenmcnaughton and she does not know how to move this forward. @mattwire do you think this PR is review ready?

@mattwire
Copy link
Contributor Author

@jaapjansma Yes I think this is review-ready. There is a list of forms in the description of this PR where reCAPTCHA is loaded and these have all been tested "manually" to ensure that there is no change if this PR is merged.
The overall goal was to move the reCAPTCHA code out of CiviCRM core and into an extension. This PR does not complete that process but does move most of the code across and gets us into a state where you could disable the "core extension" and replace it with another extension that implements reCAPTCHA - without this PR it's not possible to do that because you might end up with 2 reCAPTCHAs!

There are various concerns about reCAPTCHA in core that have been brought up:

  1. That it is a proprietary service and it causes problems for some organisations.
  2. That it is not loaded consistently across different forms (eg. some might check if user is logged in but others might not).
  3. That the configuration in CiviCRM to control if we should load reCAPTCHA is not very well designed (eg. civicrm_uf_group has a field is_captcha but this is not read directly, instead it is copied to all fields in the profile and then checked there). It would be better to have an independent mechanism to control on which pages the reCAPTCHA should load.
  4. There are patent concerns surrounding reCAPTCHAv2 which some people have raised concerns to me about. So it would be better not in core.

To share my goal, I have a Work in Progress branch for https://lab.civicrm.org/extensions/formprotection which already contains logic for controlling which forms we should load reCAPTCHA (independent of the existing database tables). It will allow selection of reCAPTCHAv2 or v3.

@BettyDolfing
Copy link

Test this please

@jaapjansma
Copy link
Contributor

@mattwire the test failed. Can you have a look?


Stacktrace

CRM_Profile_Form_EditTest::testProfileUrl
Trying to access array offset on value of type null

/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/ext/recaptcha/CRM/Utils/ReCAPTCHA.php:186
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/ext/recaptcha/recaptcha.php:121
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/CRM/Utils/Hook.php:275
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/CRM/Utils/Hook/UnitTests.php:99
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/Civi/Core/CiviEventDispatcher.php:241
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/EventDispatcher.php:214
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/vendor/symfony/event-dispatcher/EventDispatcher.php:44
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/Civi/Core/CiviEventDispatcher.php:198
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/CRM/Utils/Hook.php:167
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/CRM/Utils/Hook.php:471
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/CRM/Core/Form.php:701
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/tests/phpunit/CRM/Profile/Form/EditTest.php:44
/home/jenkins/bknix-dfl/build/core-20588-bnpc/web/sites/all/modules/civicrm/tests/phpunit/CiviTest/CiviUnitTestCase.php:262
/home/jenkins/bknix-dfl/extern/phpunit8/phpunit8.phar:671

@mattwire mattwire force-pushed the recaptcharefactor branch from 7a1af4a to c1b477a Compare May 5, 2022 20:13
@jaapjansma
Copy link
Contributor

@mattwire did you change anything? Is this PR ready for review?

@mattwire
Copy link
Contributor Author

@jaapjansma Yes I fixed the test failures. In my opinion it is ready for review

@BettyDolfing
Copy link

Test this please

@jaapjansma
Copy link
Contributor

  • General standards
    • Explain (r-explain)
      • PASS : The goal/problem/solution have been adequately explained in the PR.
    • User impact (r-user)
      • PASS: The change would be unnoticeable for a majority of users who work with this feature.
    • Documentation (r-doc)
      • PASS: The changes do not require documentation.
    • Run it (r-run)
      • PASS: We created a recaptcha key in Google then added that key to civicrm. Used a profile with recaptcha enabled and tried this profile. We did the same for a contribution page.
  • Developer standards
    • Technical impact (r-tech)
      • PASS: The change potentially affects compatibility, but the risks have been sufficiently managed.
      • COMMENTS: The risks have been managed because the extension is shipped with core.
    • Code quality (r-code)
      • PASS: The functionality, purpose, and style of the code seems clear+sensible.
    • Maintainability (r-maint)
      • PASS: The change sufficiently improves test coverage, or the change is trivial enough that it does not require tests.
      • COMMENTS: Putting this in an extension makes it easier to move to another recaptcha provider when needed in the future.
    • Test results (r-test)
      • PASS: The test results are all-clear.

@eileenmcnaughton or @colemanw can one of you merge this PR?

@jaapjansma jaapjansma added the merge ready PR will be merged after a few days if there are no objections label May 23, 2022
@demeritcowboy
Copy link
Contributor

jenkins retest this please

@demeritcowboy demeritcowboy merged commit 9f92cd4 into civicrm:master Jul 10, 2022
@mattwire mattwire deleted the recaptcharefactor branch July 10, 2022 21:15
@mattwire
Copy link
Contributor Author

Ah, thanks @demeritcowboy :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
master merge ready PR will be merged after a few days if there are no objections
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants