-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Issue/279 code upload #545
Merged
Merged
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
948426d
added additional validation to files api
wouterdb ed65d5a
changed code upload to new storage mechanism
wouterdb 4eb2015
added batched upload api
wouterdb 57899f0
basic working version of batched code upload, temp commit
wouterdb e7d11d5
fixed pep 8
wouterdb 0308cef
Merge branch 'master' into issue/279-code-upload
bartv 0329130
Rewrite changelog and help message
bartv File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -16,7 +16,6 @@ | |
Contact: [email protected] | ||
""" | ||
|
||
import hashlib | ||
import logging | ||
import os | ||
import time | ||
|
@@ -32,6 +31,7 @@ | |
from inmanta.ast import RuntimeException, CompilerException | ||
from tornado.ioloop import IOLoop | ||
from tornado import gen | ||
from inmanta.util import hash_file | ||
|
||
LOGGER = logging.getLogger(__name__) | ||
|
||
|
@@ -60,6 +60,29 @@ def __str__(self, *args, **kwargs): | |
return "Cycle in dependencies: %s" % self.cycle | ||
|
||
|
||
@gen.coroutine | ||
def upload_code(conn, tid, version, resource_to_sourcemap): | ||
allfiles = {myhash: source_code for sourcemap in resource_to_sourcemap.values() | ||
for myhash, (file_name, module, source_code, req) in sourcemap.items()} | ||
|
||
res = yield conn.stat_files(list(allfiles.keys())) | ||
if res is None or res.code != 200: | ||
raise Exception("Unable to upload handler plugin code to the server (msg: %s)" % res.result) | ||
|
||
for file in res.result["files"]: | ||
res = yield conn.upload_file(id=file, content=base64.b64encode(allfiles[file].encode()).decode("ascii")) | ||
if res is None or res.code != 200: | ||
raise Exception("Unable to upload handler plugin code to the server (msg: %s)" % res.result) | ||
|
||
compactmap = {resource: {myhash: (file_name, module, req) for | ||
myhash, (file_name, module, source_code, req)in sourcemap.items()} | ||
for resource, sourcemap in resource_to_sourcemap.items()} | ||
|
||
res = yield conn.upload_code_batched(tid=tid, id=version, resources=compactmap) | ||
if res is None or res.code != 200: | ||
raise Exception("Unable to upload handler plugin code to the server (msg: %s)" % res.result) | ||
|
||
|
||
class Exporter(object): | ||
""" | ||
This class handles exporting the compiled configuration model | ||
|
@@ -354,10 +377,7 @@ def merge_dict(a, b): | |
|
||
@gen.coroutine | ||
def call(): | ||
for myresource, mysources in sources.items(): | ||
res = yield conn.upload_code(tid=tid, id=version, resource=myresource, sources=mysources) | ||
if res is None or res.code != 200: | ||
raise Exception("Unable to upload handler plugin code to the server (msg: %s)" % res.result) | ||
yield upload_code(conn, tid, version, sources) | ||
|
||
self.run_sync(call) | ||
|
||
|
@@ -431,15 +451,6 @@ def get_unknown_resources(self, hostname): | |
|
||
return set() | ||
|
||
def _hash_file(self, content): | ||
""" | ||
Create a hash from the given content | ||
""" | ||
sha1sum = hashlib.new("sha1") | ||
sha1sum.update(content) | ||
|
||
return sha1sum.hexdigest() | ||
|
||
def upload_file(self, content=None): | ||
""" | ||
Upload a file to the configuration server. This operation is not | ||
|
@@ -448,7 +459,7 @@ def upload_file(self, content=None): | |
if not isinstance(content, bytes): | ||
content = content.encode('utf-8') | ||
|
||
hash_id = self._hash_file(content) | ||
hash_id = hash_file(content) | ||
self._file_store[hash_id] = content | ||
|
||
return hash_id | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -98,6 +98,9 @@ def validate_fact_renew(value): | |
agent_timeout = Option("server", "agent-timeout", 30, | ||
"Time before an agent is considered to be offline", is_time) | ||
|
||
server_delete_currupt_files = Option("server", "delete_currupt_files", True, | ||
""" When the server finds corrupt files, delete them automatically""", is_bool) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sounds a bit scary. I would prefer a help text like this: |
||
|
||
############################# | ||
# Dashboard | ||
############################# | ||
|
@@ -116,5 +119,6 @@ def default_hangtime(): | |
""" server.agent-timeout*3/4 """ | ||
return str(int(agent_timeout.get() * 3 / 4)) | ||
|
||
|
||
agent_hangtime = Option("server", "agent-hold", default_hangtime, | ||
"Maximal time the server will hold an agent heartbeat call", is_time) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Internal storage format for code is optimized. This introduces API and schema changes. This release supports both storage versions. The old version will be removed in the next release.