Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FastaInputFormat splits file into too many partitions #207

Open
PatrycjaKarbownik opened this issue Jan 18, 2020 · 0 comments
Open

FastaInputFormat splits file into too many partitions #207

PatrycjaKarbownik opened this issue Jan 18, 2020 · 0 comments

Comments

@PatrycjaKarbownik
Copy link

PatrycjaKarbownik commented Jan 18, 2020 •
edited
Loading

I used the FastaInputFormat to read fasta file in Spark.
After code analysis, I see that getSplits method is overrided. Okey, but number of returned partitions equals number of fasta records.
E.g: fasta file has 661 bytes (13 fasta records) and my mapred.max.split.size conf is set to 500 bytes. I expect to get 2 partitions, but I get 13 partitions. What is purpose for that extra splitting? Why getSplits method from FileInputFormat couldn't be used?

Moreover FastqInputFormat uses only getSplits method from FileInputFormat and it's behaviour is expected by me.
@PatrycjaKarbownik PatrycjaKarbownik changed the title FastaInputReader splits file to too many partitions FastaInputReader splits file into too many partitions Jan 18, 2020
@PatrycjaKarbownik PatrycjaKarbownik changed the title FastaInputReader splits file into too many partitions FastaInputFormat splits file into too many partitions Jan 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@PatrycjaKarbownik