-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Impressive Speedup using activerecord-import #66
Comments
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Jun 19, 2018
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Jul 20, 2018
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Mar 20, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 16, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 16, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 16, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 16, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 16, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 25, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Oct 28, 2019
aiomaster
added a commit
to webit-de/data-anonymization
that referenced
this issue
Jun 7, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have very large tables that I want to anonymize.
A simple run of the anonymization code took me near to 40 minutes!
So I tried to optimize the code a little and could get it down to 5 minutes by using the activerecord-import gem.
I update my records on a postgresql 10 database using the Blacklist strategy.
The trick is to not save every single record, but collect them and use the import-method of activerecord-import with its On-Duplicate-Key-Update-Strategy. Problem is, that it just works for mysql and postgresql that way.
To test this just add the gem 'activerecord-import' use my fork and run the anonymization against a mysql or postgresql database.
Maybe I can make a pull request, but I have just tested my own case and don't know if something else is broken.
Are you interested in such a feature?
The text was updated successfully, but these errors were encountered: