Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

File writer/sender doesn't fsync #284

Open
KristianLyng opened this issue Jan 5, 2023 · 0 comments
Open

File writer/sender doesn't fsync #284

KristianLyng opened this issue Jan 5, 2023 · 0 comments
Labels
bug Something isn't working sender Sender-related issue

Comments

@KristianLyng
Copy link
Collaborator

When writing, the file sender doesn't issue a fsync, which means it will return before data is comitted to disk. This isn't normally a problem, but with a one-off receiver chain it easily leads to incomplete writes.

E.g.:

{
    "receivers": {
        "dbpoll": {
            "type": "sql",
            "connstr": "...",
            "driver": "mysql",
            "handler": "dbp",
            "query": "select .... (large query)"
        }
    },
    "handlers": {
        "dbp": {
            "parser": "json",
            "transformers": ["now"],
            "sender": "tofile"
        }
    },
    "senders": {
        "tofile": {
            "type": "file",
            "encoder": "skogul",
            "path": "dump.json"
        }
    }
}

This will break if the query returns large amounts of data, since the receiver will return and skogul will exit before data is committed. Or at least, I am assuming that's the reason.

@KristianLyng KristianLyng added bug Something isn't working sender Sender-related issue labels Jan 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working sender Sender-related issue
Projects
None yet
Development

No branches or pull requests

1 participant