Matomo ldap sync users upper limit is 999 #306

lizzyliao · 2022-10-05T07:01:25Z

Hi All:
matomo ldap sync users upper limit is 999,
I can not sync any ldap users into matomo,

How do I solve this problem ?
Thank you very much.

Matomo Version: 4.7.1
PHP Version: PHP 7.4.30
Server Operating System: centos7
Additionally installed plugins: ldap

The text was updated successfully, but these errors were encountered:

AltamashShaikh · 2022-10-05T07:26:52Z

@lizzyliao what error are you getting ?
And are you using the ./console loginldap:synchronize-users command to sync users ?

lizzyliao · 2022-10-05T08:38:46Z

Hi AltamashShaikh :
Yes, use ./console loginldap:synchronize-users command.

AltamashShaikh · 2022-10-05T09:27:11Z

@lizzyliao What error do you get ?

lizzyliao · 2022-10-06T00:28:33Z

@AltamashShaikh
No error message shown. Sync successfully.
But just only "Synchronized 999 users!"
Thank you very much~

AltamashShaikh · 2022-10-06T01:54:41Z

@lizzyliao I do not see any hard limits in code, may be you can try running the command using ./console loginldap:synchronize-users -vvv and see if there is any error message or any other debug message helpful for us to debug this issue.

lizzyliao · 2022-10-06T02:30:10Z

@AltamashShaikh
We see error messasge as following:
"Synchronized 999 users!
Could not synchronize the following users in LDAP:
K21050416 Could not instantiate mail function."

But we can see 'K21050416" already in matomo web ui.
Administration -> System -> users
**The users page show 'K21050416' with correct email "[email protected]" **

The program stopped and some other users can not sync into matomo.

Thank you very much

lizzyliao · 2022-10-06T02:31:12Z

@AltamashShaikh

we use debug flag log as following:
DEBUG [2022-10-06 02:28:31] 26238 ldap_search result is [resource]
DEBUG [2022-10-06 02:28:31] 26238 Calling ldap_get_entries([resource], [resource])
DEBUG [2022-10-06 02:28:31] 26238 ldap_get_entries result is not null
DEBUG [2022-10-06 02:28:31] 26238 Calling ldap_close([resource])
DEBUG [2022-10-06 02:28:31] 26238 ldap_close returned true
DEBUG [2022-10-06 02:28:31] 26238 Model\LdapUsers: end getUser() with array["objectclass","cn","sn","descriptio n","distinguishedname","instancetype","whencreated","whenchanged","displayname","usncreated","info","memberof"," usnchanged","proxyaddresses","homemdb","submissioncontlength","garbagecollperiod","mdbusedefaults","mailnickname ","protocolsettings","internetencoding","name","objectguid","useraccountcontrol","pwdlastset","primarygroupid"," objectsid","samaccountname","samaccounttype","showinaddressbook","legacyexchangedn","userprincipalname","objectc ategory","dscorepropagationdata","lastlogontimestamp","mail","thumbnailphoto","msexchpoliciesexcluded","msexchom aadminwirelessenable","msexchhomeservername","msexchmailboxsecuritydescriptor","msexchuseraccountcontrol","msexc hmailboxguid","msexchmailboxfolderset","msexchtransportrecipientsettingsflags","msexchumdtmfmap","msexchmdbrules quota","msexchaddressbookflags","msexchprovisioningflags","msexchmailboxtemplatelink","msexchumenabledflags2","m sexchwhenmailboxcreated","msexchrecipientdisplaytype","msexchmailboxauditenable","msexchrbacpolicylink","msexchr ecipientsoftdeletedstatus","msexchcalendarloggingquota","msexchversion","msexchmailboxauditlogagelimit","msexchr ecipienttypedetails","msexchdumpsterquota","msds-externaldirectoryobjectid","msexchdumpsterwarningquota","msexch moderationflags","msexcharchivequota","msexcharchivewarnquota","msexchelcmailboxflags","msexchbypassaudit","msex chtextmessagingstate","msexchgroupsecurityflags","dn"]
DEBUG [2022-10-06 02:28:31] 26238 UserSynchronizer::synchronizeLdapUser: synchronizing user [ piwik login = Wit s.KevinHuang, ldap login = Wits.KevinHuang ]
success!

Synchronized 1000 users!

Error: error or warning logs detected, exit 1

AltamashShaikh · 2022-10-06T04:04:59Z

@AltamashShaikh We see error messasge as following: "Synchronized 999 users! Could not synchronize the following users in LDAP: K21050416 Could not instantiate mail function."

But we can see 'K21050416" already in matomo web ui. Administration -> System -> users **The users page show 'K21050416' with correct email "[email protected]" **

The program stopped and some other users can not sync into matomo.

Thank you very much

The error "Could not instantiate mail function" is a error from PHPMailer, can you check if you have setup the SMTP server correctly ?

lizzyliao · 2022-10-06T06:39:34Z

@AltamashShaikh
We have never set up an smtp server.
Will this affect sync ldap users?

I type "./console loginldap:synchronize-users" command again.
The error message "K21050416 Could not instantiate mail function" do not show again.
The error message is as following:
_

"EBUG [2022-10-06 06:19:52] 126938 ldap_bind result is '1'
DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_search([resource], 'dc=xxxx', '(&(&(objectClass=Person)(memb erOf:1.2.840.113556.1.4.1941:=cn=matomopaas,ou=Group_Object,dc=wih,dc=xxxx) )(samaccountname=Wits.KevinHuang) )')
DEBUG [2022-10-06 06:19:52] 126938 ldap_search result is [resource]
DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_get_entries([resource], [resource])
DEBUG [2022-10-06 06:19:52] 126938 ldap_get_entries result is not null
DEBUG [2022-10-06 06:19:52] 126938 Calling ldap_close([resource])
DEBUG [2022-10-06 06:19:52] 126938 ldap_close returned true
DEBUG [2022-10-06 06:19:52] 126938 Model\LdapUsers: end getUser() with array["objectclass","cn","sn","descripti on","distinguishedname","instancetype","whencreated","whenchanged","displayname","usncreated","info","memberof", "usnchanged","proxyaddresses","homemdb","submissioncontlength","garbagecollperiod","mdbusedefaults","mailnicknam e","protocolsettings","internetencoding","name","objectguid","useraccountcontrol","pwdlastset","primarygroupid", "objectsid","samaccountname","samaccounttype","showinaddressbook","legacyexchangedn","userprincipalname","object category","dscorepropagationdata","lastlogontimestamp","mail","thumbnailphoto","msexchpoliciesexcluded","msexcho maadminwirelessenable","msexchhomeservername","msexchmailboxsecuritydescriptor","msexchuseraccountcontrol","msex chmailboxguid","msexchmailboxfolderset","msexchtransportrecipientsettingsflags","msexchumdtmfmap","msexchmdbrule squota","msexchaddressbookflags","msexchprovisioningflags","msexchmailboxtemplatelink","msexchumenabledflags2"," msexchwhenmailboxcreated","msexchrecipientdisplaytype","msexchmailboxauditenable","msexchrbacpolicylink","msexch recipientsoftdeletedstatus","msexchcalendarloggingquota","msexchversion","msexchmailboxauditlogagelimit","msexch recipienttypedetails","msexchdumpsterquota","msds-externaldirectoryobjectid","msexchdumpsterwarningquota","msexc hmoderationflags","msexcharchivequota","msexcharchivewarnquota","msexchelcmailboxflags","msexchbypassaudit","mse xchtextmessagingstate","msexchgroupsecurityflags","dn"]
DEBUG [2022-10-06 06:19:52] 126938 UserSynchronizer::synchronizeLdapUser: synchronizing user [ piwik login = Wi ts.KevinHuang, ldap login = Wits.KevinHuang ]
success!

Synchronized 1000 users!

Error: error or warning logs detected, exit 1"

/var/log/cron

Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126575]: starting 0anacron
Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126584]: finished 0anacron
Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126575]: starting mcelog.cron
Oct 6 14:01:01 matomo-dev run-parts(/etc/cron.hourly)[126590]: finished mcelog.cron
Oct 6 14:10:01 matomo-dev CROND[126749]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 6 14:10:01 matomo-dev CROND[126751]: (root) CMD (/matomo/matomo/console loginldap:synchronize-users)
Oct 6 14:10:26 matomo-dev CROND[126748]: (root) MAIL (mailed 153215 bytes of output but got status 0x004b#012)
Oct 6 14:20:01 matomo-dev CROND[126941]: (root) CMD (/matomo/matomo/console loginldap:synchronize-users)
Oct 6 14:20:01 matomo-dev CROND[126942]: (root) CMD (/usr/lib64/sa/sa1 1 1)
Oct 6 14:20:34 matomo-dev CROND[126940]: (root) MAIL (mailed 153215 bytes of output but got status 0x004b#012)

Thank you very much~

AltamashShaikh · 2022-10-06T08:06:44Z

@lizzyliao Thanks for the log, I will check more on this and for now no need to update/change anything.
I will get back to you on this.

AltamashShaikh · 2022-10-06T08:10:46Z

@lizzyliao How many users do you have for sync ?

lizzyliao · 2022-10-07T02:31:31Z

@AltamashShaikh
I think maybe about 1100 people, but the number will increase over time.

Thank you very much.

AltamashShaikh · 2022-10-07T03:32:14Z

@lizzyliao When you re run it does it sync all the 1100 or is it stopping at 1000 only ?

lizzyliao · 2022-10-07T05:29:18Z

@AltamashShaikh
The matomo server is stopping at 1000 only with error messages
"Error: error or warning logs detected, exit 1"

Thank you very much.

AltamashShaikh · 2022-10-07T06:19:16Z

@lizzyliao Ill try to remove my SMTP settings and try to sync a new user and will see if it creates any issue

sgiehl · 2022-10-07T07:46:10Z

@lizzyliao @AltamashShaikh might that be an issue of the ldap server? I actually haven't done much with ldap the last years, but if I remember correctly there was some sort of "security" policy to limit the number of results. Was is called maxpagesize or so? 🤔

AltamashShaikh · 2022-10-07T08:43:21Z

@sgiehl Thanks I was not aware of this limit and thought it was exiting due to mailer settings.
Its indeed a security feature by LDAP to prevent DDOS.
@lizzyliao @sgiehl is correct there is a limit
Refer this blog article which explains why its being limited to 1000

AltamashShaikh · 2022-10-10T02:06:11Z

@lizzyliao Is there anything you need help with it ? Or should we close this issue ?

lizzyliao · 2022-10-12T23:53:04Z

@sgiehl @AltamashShaikh
Thank you for your information, I will contact our ldap server admin.
Thank you very much.

lizzyliao · 2022-10-14T07:02:03Z

@AltamashShaikh
Hi AltamashShaikh
(1) Our ldap admin say ldap query default = 1000, but you can query more than 1000 users when you query ldap.
(2) I modify plugins/LoginLdap/Ldap/Client.php in matomo system, I add three lines as following:
It works. It only can sync 100 users. But set pageSize =2000, it only sync 1000 users.

  $pageSize = 100;
  $cookie = '';        
  ldap_control_paged_result($connectionResource, $pageSize, true, $cookie);
  $result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes);

(3) I think If we need sync more than 1000 users, we maybe need use while loop ,
please see the following url content.
https://stackoverflow.com/questions/8636375/php-ldap-search-size-limit-exceeded

 Thank you very much

AltamashShaikh · 2022-10-14T07:25:35Z

Hey @lizzyliao
Thanks for checking and posting above solution, but ldap_control_paged_result is deprecated in PHP 7.4 and removed in PHP 8.0.
I will check what other alternatives we have, can you update the LDAP server setting at your end to fetch more than 1000 records ?
We can latter set the sizeLimit as 0 to fetch all the records - Refer https://www.php.net/manual/en/function.ldap-search.php

We need to replace this line https://github.com/matomo-org/plugin-LoginLdap/blob/4.x-dev/Ldap/Client.php#L382
with below line

$result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes, $attributes_only = 0, $sizelimit = 0);

AltamashShaikh · 2022-10-14T07:27:50Z

@lizzyliao If you are gonna try above fix, you need to

You need to update your LDAP server to return more than 1000 records
Replace the above line as suggested and try and it should work.

lizzyliao · 2022-10-14T08:31:56Z

@AltamashShaikh
I write this program for testing. I can sync 1349 users.

$i=0;
 do {
                $result  = ldap_search($conn, $dn, $filter, $justthese,0,-1,0,LDAP_DEREF_NEVER,[['oid' => LDAP_CONTROL_PAGEDRESULTS, 'value' => ['size' => 1000, 'cookie' => $cookie]]]);
                ldap_parse_result($conn, $result, $errcode , $matcheddn , $errmsg , $referrals, $controls);
                $entries = ldap_get_entries($conn, $result);
            foreach ($entries as $entry) {
                echo "cn: ".$entry['cn'][0]."\n";
                $i++;
            }
            if (isset($controls[LDAP_CONTROL_PAGEDRESULTS]['value']['cookie'])) {
                // Vous devez passer le cookie du dernier appel au prochain
                $cookie = $controls[LDAP_CONTROL_PAGEDRESULTS]['value']['cookie'];
            } else {
                $cookie = '';
            }

 } while (!empty($cookie));

 echo "i= ".$i

I think need use this parameter "LDAP_CONTROL_PAGEDRESULTS"

lizzyliao · 2022-10-14T09:02:37Z

@AltamashShaikh
I modify Client.php, it still only can sync 1000 users.
$result = ldap_search($connectionResource, $baseDn, $ldapFilter, $attributes, $attributes_only = 0, $sizelimit = 0);
I use parameter "LDAP_CONTROL_PAGEDRESULTS" and cookie , It can sync more than 1000 users in my testing program.
But I modify Client.php according to my testing program, It can not work.
I need your help~~
Thank you very much.

AltamashShaikh · 2022-10-14T09:05:33Z

Okay, I would check the above code you shared if we can add that and won't create any regressions, but this will take time as I need to prioritize it

lizzyliao · 2022-10-14T09:07:29Z

@AltamashShaikh

OK, Thank you very much. :)

AltamashShaikh · 2022-10-14T09:14:34Z

@lizzyliao just 1 Q did you update the limit from 1000 to your desired number in your LDAP server?

lizzyliao · 2022-10-17T00:03:36Z

@AltamashShaikh
NO, We don't do any change in our LDAP server.

AltamashShaikh · 2022-10-17T01:27:19Z

@lizzyliao I have added this task for prioritisation, so that our product team can put it into existing workload

lizzyliao · 2022-10-17T02:47:45Z

@AltamashShaikh
Thank you very much.

heurteph-ei · 2023-06-02T08:55:37Z

Hi @mattab, @sgiehl, @AltamashShaikh
Is it possible to remove the label "answered" if this ticket, as there is still things to do to complete it?
I think also that increase the number of LDAP responses is not a good solution, as this can downgrade the LDAP security. Do you know when you'll be able to work on this ticket?

AltamashShaikh · 2023-06-05T01:26:59Z

@heurteph-ei We currently haven't scheduled this one as we have other priority items scheduled, will update here once we have any update.

heurteph-ei · 2023-11-13T09:32:22Z

Hi @mattab, @sgiehl, @AltamashShaikh
Is it possible to remove the label "answered" if this ticket, as there is still things to do to complete it?

AltamashShaikh · 2023-11-14T04:02:24Z

@heurteph-ei updated 👍

lizzyliao added the Potential Bug label Oct 5, 2022

AltamashShaikh transferred this issue from matomo-org/matomo Oct 5, 2022

AltamashShaikh closed this as completed Oct 13, 2022

AltamashShaikh added answered and removed Potential Bug labels Oct 13, 2022

AltamashShaikh reopened this Oct 14, 2022

AltamashShaikh added this to the For Prioritisation milestone Oct 17, 2022

mattab modified the milestones: For Prioritisation, Priority Backlog Oct 28, 2022

heurteph-ei mentioned this issue Jun 2, 2023

Fails Synchronizing Large Directories #177

Open

AltamashShaikh removed the answered label Nov 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Matomo ldap sync users upper limit is 999 #306

Matomo ldap sync users upper limit is 999 #306

lizzyliao commented Oct 5, 2022

AltamashShaikh commented Oct 5, 2022

lizzyliao commented Oct 5, 2022

AltamashShaikh commented Oct 5, 2022

lizzyliao commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 6, 2022 •

edited

Loading

lizzyliao commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 6, 2022 •

edited

Loading

AltamashShaikh commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 7, 2022

AltamashShaikh commented Oct 7, 2022

lizzyliao commented Oct 7, 2022 •

edited

Loading

AltamashShaikh commented Oct 7, 2022

sgiehl commented Oct 7, 2022

AltamashShaikh commented Oct 7, 2022

AltamashShaikh commented Oct 10, 2022

lizzyliao commented Oct 12, 2022

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

lizzyliao commented Oct 14, 2022 •

edited

Loading

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022 •

edited

Loading

lizzyliao commented Oct 17, 2022

AltamashShaikh commented Oct 17, 2022

lizzyliao commented Oct 17, 2022

heurteph-ei commented Jun 2, 2023

AltamashShaikh commented Jun 5, 2023

heurteph-ei commented Nov 13, 2023

AltamashShaikh commented Nov 14, 2023

Matomo ldap sync users upper limit is 999 #306

Matomo ldap sync users upper limit is 999 #306

Comments

lizzyliao commented Oct 5, 2022

AltamashShaikh commented Oct 5, 2022

lizzyliao commented Oct 5, 2022

AltamashShaikh commented Oct 5, 2022

lizzyliao commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 6, 2022 • edited Loading

lizzyliao commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 6, 2022 • edited Loading

AltamashShaikh commented Oct 6, 2022

AltamashShaikh commented Oct 6, 2022

lizzyliao commented Oct 7, 2022

AltamashShaikh commented Oct 7, 2022

lizzyliao commented Oct 7, 2022 • edited Loading

AltamashShaikh commented Oct 7, 2022

sgiehl commented Oct 7, 2022

AltamashShaikh commented Oct 7, 2022

AltamashShaikh commented Oct 10, 2022

lizzyliao commented Oct 12, 2022

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

lizzyliao commented Oct 14, 2022 • edited Loading

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022

lizzyliao commented Oct 14, 2022

AltamashShaikh commented Oct 14, 2022 • edited Loading

lizzyliao commented Oct 17, 2022

AltamashShaikh commented Oct 17, 2022

lizzyliao commented Oct 17, 2022

heurteph-ei commented Jun 2, 2023

AltamashShaikh commented Jun 5, 2023

heurteph-ei commented Nov 13, 2023

AltamashShaikh commented Nov 14, 2023

lizzyliao commented Oct 6, 2022 •

edited

Loading

lizzyliao commented Oct 6, 2022 •

edited

Loading

lizzyliao commented Oct 7, 2022 •

edited

Loading

lizzyliao commented Oct 14, 2022 •

edited

Loading

AltamashShaikh commented Oct 14, 2022 •

edited

Loading