Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Impressive Speedup using activerecord-import #66

Open
aiomaster opened this issue Jun 19, 2018 · 0 comments
Open

Impressive Speedup using activerecord-import #66

aiomaster opened this issue Jun 19, 2018 · 0 comments

Comments

@aiomaster
Copy link

aiomaster commented Jun 19, 2018

I have very large tables that I want to anonymize.
A simple run of the anonymization code took me near to 40 minutes!
So I tried to optimize the code a little and could get it down to 5 minutes by using the activerecord-import gem.
I update my records on a postgresql 10 database using the Blacklist strategy.
The trick is to not save every single record, but collect them and use the import-method of activerecord-import with its On-Duplicate-Key-Update-Strategy. Problem is, that it just works for mysql and postgresql that way.
To test this just add the gem 'activerecord-import' use my fork and run the anonymization against a mysql or postgresql database.

Maybe I can make a pull request, but I have just tested my own case and don't know if something else is broken.
Are you interested in such a feature?

aiomaster added a commit to webit-de/data-anonymization that referenced this issue Jun 19, 2018
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Jul 20, 2018
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Mar 20, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 16, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 16, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 16, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 16, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 16, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 25, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Oct 28, 2019
aiomaster added a commit to webit-de/data-anonymization that referenced this issue Jun 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant