[WIP] Alterative hashings for data (#1676) #1920

vyloy · 2019-04-24T01:18:18Z

Based on branch 1654 from @efiop 's fork

Fixes #1676

efiop · 2019-04-24T18:16:34Z

@vyloy Please rebase.

efiop · 2019-04-25T18:53:52Z

@vyloy Your tests are failing.

dvc/stage.py

efiop · 2019-04-26T01:11:50Z

dvc/utils/__init__.py

    filtered = dict_filter(d, exclude)
    byts = json.dumps(filtered, sort_keys=True).encode("utf-8")
-    return bytes_md5(byts)
+    hasher = CHECKSUM_MAP[checksum_type]()


why not return bytes_checksum(byts, checksum_type) then?

efiop · 2019-04-26T01:17:58Z

dvc/stage.py

        self.locked = locked
        self.tag = tag
        self._state = state or {}

+        if repo and repo.config and repo.config.config:


this code matches the one in remote local. How about we just use repo.cache.local.hash instead, since it is pretty much the same thing and local cache is guaranteed to exist?

efiop · 2019-04-26T01:24:39Z

dvc/remote/local.py

@@ -202,8 +216,11 @@ def isdir(self, path_info):
    def walk(self, path_info):
        return os.walk(path_info["path"])

+    def get_hash_list(self):


what is the point of this wrapper?

to reuse the hash list inside output.base

So why not just set self.hash = [self.PARAM_CHECKSUM] in the RemoteBase and use it instead of get_hash_list and get_prefer_hash_type?

From my practice (in other languages), functions or methods are more flexible/optimizable/better than class fields in many cases.

The first thought was to prevent hash get changed from outside.

Like get_prefer_hash_type will be more clear and easy to refactor than self.hash[0] in the code(11 callers) in the future, and the callers don't have to figure out the differences between different remote implements, on the other hand, other remote implements are free to use other different strategies.

set self.hash = [self.PARAM_CHECKSUM] in the RemoteBase will need long-term memory in heap for gc process to check, instead of short-term or stack-alloc(if compiler is smart enough) here for other implements.

Great points! getters/setters are nice when you have some complex logic, but in our case properties are more than enough.

Thinking about it a little more, it actually feels like much of that checksum logic should be contained in the dvc/remote/local. That way you won't even have to access those hash types from the outside. For example, Stage.changed_checksum() could then look like:

return self.checksum != self._compute_checksum()

and Stage.compute_checksum() would

... return self.cache.local.dict_checksum(d, exclude=...)

that way almost all of the internals are hidden where they matter.

It is actually pretty similar to how we handle RemoteLOCAL.cache_types.

refactoring so much lines for showing properties are more than enough?

@vyloy Sorry, what do you mean?

Refactoring this two places to avoid use property self.hash = [self.PARAM_CHECKSUM] outside remote is fine, but how about other places, they need to be consistent, right?

for example stage.py is_cached:

for out in outs: out.pop(self.repo.cache.local.get_prefer_hash_type(), None) out.pop(RemoteS3.PARAM_CHECKSUM, None)

efiop · 2019-04-26T01:27:00Z

dvc/remote/local.py

+                self.hash = [utils.CHECKSUM_MD5]
+        else:
+            self.hash = [utils.CHECKSUM_MD5]
+        RemoteLOCAL.PARAM_CHECKSUM = self.hash[0]


Overwriting constants is not nice.

efiop · 2019-04-26T01:28:53Z

dvc/remote/local.py

+                if isinstance(core_hash, str):
+                    core_hash = [h.strip() for h in core_hash.split(",")]
+                self.hash = core_hash
+            else:


you can just set this once before if instead of having two elses

efiop · 2019-04-26T07:56:49Z

dvc/output/base.py

@@ -154,10 +158,14 @@ def exists(self):
        return self.remote.exists(self.path_info)

    def changed_checksum(self):
+        cs = self.checksum
+        if cs is None:
+            self.remote.update_state_checksum(self.path_info)


why is this needed? state is updated in get_file_checksum and get_dir_checksum already.

this fixes a bug while we use unit test and terminal in the same time.
if we initial the environment by test(e.g. add foo), and then change the config to sha256 in terminal, then it shows that it will uses the hash from state in the dvc file directly without rehash(e.g. in the dvc file, out: sha256: xxxxxx(this is a md5)).

I don't think this is a proper way to solve this. We should have a way to distinguish between different checksums in the state file itself. Seems like introducing a column for each checksum type would be a good way to go.

It is a quick hack.
if cs is None, means config hash.local has been changed and no record fits, same to state record, so this update to state is needed anyway.

for example, changing config hash.local from md5 to sha256 will go into this condition, while from md5 to sha256, md5 don't.

Introducing a column for each checksum type, many places need to be changed, and will lead to worse performance because additional checks that do not necessary for usual use cases.

For example in the unit test:

def test_incompatibility(self): ret = main(["add", self.FOO]) self.assertEqual(0, ret) ret = main(["config", "hash.local", "sha256"]) self.assertEqual(0, ret) ret = main(["status"]) self.assertEqual(0, ret) ret = main(["status", "-q"]) self.assertEqual(1, ret) ret = main(["add", self.FOO]) // update_state_checksum is called here. self.assertEqual(0, ret)

I still don't get your point...
Why self.remote.save_info can expose while self.remote.update_state_checksum can not?

For example?

The fact that this particular output doesn't have some checksum doesn't mean that it is not used anywhere else.

Why self.remote.save_info can expose while self.remote.update_state_checksum can not?

Because save_info is not exposing state notion, while update_state_checksum does. Your update_state_checksum is almost a full copy of get_checksum and it is called right before it, which is just not right. What we could do instead is leave checksum type notion inside of the Remote, so it is the one dealing with it without you having to call dirty hacks from Output. For example, you could get rid of update_state_checksum if you simply make Remote.get_checksum do smth like checksum = self.state.get(path_info, checksum_type=self.checksum), so if it say doesn't have sha256, then it would return None. This would help us avoid a lot of code duplication.

For example?

For example, you have two branches: one is using sha256 and the other old one is still using md5s. We could also decide to allow per-output/dependency checksum type configuration in the future, so the same file could be used with different checksums. So it makes sense to make this generic enough.

If the name is exposing state notion, how about change the name to refresh_info?

It will return True then, why do you say so? And it is only called while the output checksum is not match with config. That is not even the usual use case.

it is called right before it, which is just not right.

If we leave checksum type notion inside of the Remote, how would Remote know what checksum type is output using for querying from state to compare?

Actually, I have tried this before used this most simple way, this way leads to a lot of code refactoring. 🙂 Why do people always like to ignore the simplest and straightforward way?

For example, you could get rid of update_state_checksum if you simply make Remote.get_checksum do smth like checksum = self.state.get(path_info, checksum_type=self.checksum), so if it say doesn't have sha256, then it would return None.

IMHO, There will be a lot of advantages with this little code duplication here, e.g. better performance, less cognitive complexity etc. 🙂

That make sense from the example.

I will try some refactoring to implement this.

That is a great point! Indeed, creating columns for each additional checksum type is a bad solution. On the other hand, we still might want to keep a few checksum for each file. So currently we store md5 in md5 column, how about we replace it with checksum column instead, that would store a dict(e.g. serilized as json) instead that looks smth like {"md5": "79054025255fb1a26e4bc422aef54eb4", "sha256": "b8423365e1e2245851689f675dab11574532c71bfe2e3e53aa10fa0158e28e2b"}? It is the same way we treat checksum dicts in other places in the code.

@vyloy

Great points!

If the name is exposing state notion, how about change the name to refresh_info?

The fact still stays the same: you are exposing things that should be handled by save_info inside the remote to the outside.

It will return True then, why do you say so? And it is only called while the output checksum is not match with config. That is not even the usual use case.

Deciding whether or not state contains something valid and needs or doesn't need a refresh based on the config is pretty fragile and will only work while everyone is accessing it through changed_checksum, which might not always be the case. It is the remote itself who should decide to refresh anything if it needs to, otherwise, we are breaking the non-formal API of the Remote.

If we leave checksum type notion inside of the Remote, how would Remote know what checksum type is output using for querying from state to compare?

By using return self.remote.changed(self.path_info, self.info) in output.changed_checksum() instead of comparing self.info with self.remote.save_info by-hand 🙂

Actually, I have tried this before used this most simple way, this way leads to a lot of code refactoring. slightly_smiling_face Why do people always like to ignore the simplest and straightforward way?

Because "the simplest and straightforward" doesn't always mean "the best" 🙂 We are playing a long game here, so we need the design to make sense not only now but also in the future.

I will try some refactoring to implement this.

👍 Thanks! Looking forward to that! 🙂

efiop · 2019-04-26T08:05:23Z

dvc/remote/local.py

@@ -202,8 +216,11 @@ def isdir(self, path_info):
    def walk(self, path_info):
        return os.walk(path_info["path"])

+    def get_hash_list(self):


So why not just set self.hash = [self.PARAM_CHECKSUM] in the RemoteBase and use it instead of get_hash_list and get_prefer_hash_type?

dvc/config.py

shcheklein · 2019-04-27T00:37:33Z

dvc/output/__init__.py

@@ -41,7 +41,8 @@
 # so when a few types of outputs share the same name, we only need
 # specify it once.
 CHECKSUM_SCHEMA = {
-    schema.Optional(RemoteLOCAL.PARAM_CHECKSUM): schema.Or(str, None),
+    schema.Optional(utils.CHECKSUM_MD5): schema.Or(str, None),


should we move hashing related stuff to a separate module? so, that it looks like hash.CHECKSUM_MD5. utils should contain really generic stuff, everything else should be moving into packages with some meaningful names

dvc/remote/local.py

shcheklein · 2019-04-27T00:52:18Z

dvc/remote/local.py

+            if hash_local:
+                if isinstance(hash_local, str):
+                    hash_local = [h.strip() for h in hash_local.split(",")]
+                self.hash = hash_local


do we check for supported hashes here? It looks like a code duplicate from the config class that reads a list. Should be abstracted and reused most likely.

dvc/stage.py

shcheklein · 2019-04-27T00:56:20Z

dvc/stage.py

@@ -120,6 +120,7 @@ class Stage(object):
    STAGE_FILE_SUFFIX = ".dvc"

    PARAM_MD5 = "md5"
+    PARAM_SHA256 = "sha256"


how will we support additional checksums that were asked in the ticket? Adding them one by one like this is not scalable, it will be hard to read and extend it.

shcheklein

@vyloy good stuff! I left a few questions in the PR. To summarize:

Could you clarify why do we need to support a list of checksums per remote?
What will happen in the future if we have to checksum implementations that returns the same length keys? Should we append distinguish checksums by their name? (keep only md5 as a defautl without md5)?

In general it seems that it would be hard to support other checksums requested in the ticket (I'm even not sure why it's limited to sha256 only now). We need to abstract it in a way that it's easy to add a few implementations without touching 20 files manually.

vyloy · 2019-04-29T00:38:09Z

What are your opinions @efiop @shcheklein about the new format for hash of dvc file?

hash: 25ce158390f1477e1e418f54e445cb7f
hash-type: md5
cmd: python code.py foo file1
wdir: .
deps:
- hash: acbd18db4cc2f85cedef654fccc4a4d8
  hash-type: md5
  path: foo
- hash: d05447644b89960913c7eee5fd776adb
  hash-type: md5
  path: code.py
outs:
- hash: acbd18db4cc2f85cedef654fccc4a4d8
  hash-type: md5
  path: file1
  cache: true
  metric: false
  persist: false

if we use hash algorithm as key name, then we have to modify several files every time we add one more, e.g. schemes files.

efiop · 2019-04-29T02:46:15Z

@vyloy

- hash: acbd18db4cc2f85cedef654fccc4a4d8
  hash-type: md5

Great idea! But I'm not so sure about this format.

It is not backward compatible;
It consists of 2 lines, so merging these files would be even more of a nightmare;
I'm not sure if we would ever need it, but the current explicit format allows for having multiple checksums like so:

    md5: acbd18db4cc2f85cedef654fccc4a4d8
    sha256: cf47b5de0e349888e655572b8f6e01969661d029ec69972ef51fd0fb88f46c6a

With hash-type format, it would have to potentially look something like:

- hashes:
   - hash: acbd18db4cc2f85cedef654fccc4a4d8
     hash-type: md5
   - hash: cf47b5de0e349888e655572b8f6e01969661d029ec69972ef51fd0fb88f46c6a
     hash-type: sha256

which doesn't look nice IMHO.

if we use hash algorithm as key name, then we have to modify several files every time we add one more, e.g. schemes files.

Not really. If we would to define SCHEMA for checksums in one place, it should be pretty easy and compact to extend. Also, as @shcheklein noted, you are currently passing md5=..., sha256=... to Stage.init, which is not scalable. Maybe we could pass something like a dict hash={"md5":..., "sha256":...} instead? Or maybe there is a better way to solve this.

efiop · 2019-04-29T02:49:46Z

Regarding

hash:
    md5: acbd18db4cc2f85cedef654fccc4a4d8
    sha256: cf47b5de0e349888e655572b8f6e01969661d029ec69972ef51fd0fb88f46c6a

format: In most cases we will only have one checksum, so it would only introdue additional level of indentation

deps:
- hash:
    md5: acbd18db4cc2f85cedef654fccc4a4d8
  path: foo

which also makes it harder to read or to merge if needed.

shcheklein · 2019-04-29T02:57:56Z

@efiop @vyloy 👍on Ruslan's points and please forward on, let's keep all discussions open - either here or #dev-general on Discord.

vyloy · 2019-04-29T02:59:49Z

Got it, thanks for your helps. Especially @efiop 's great advice.

Suor · 2019-04-30T04:05:01Z

I think we shouldn't complicate .dvc file format even if that helps us with implementation. Code is volatile, API will stay. So I am in favor of simply adding keys for new hash funcs at the same level.

This is another argument to augment @efiop points.

dvc/stage.py

dvc/utils/hash/__init__.py

dvc/stage.py

dvc/config.py

vyloy · 2019-05-02T08:53:32Z

The backward compatibility will be break if just modifying state table column md5 to checksum and using json string for multiple checksum types.

Does some additional code to keep old state data working sound good?

efiop · 2019-05-02T10:06:05Z

Does some additional code to keep old state data working sound good?

Sure, but if it is too much hustle, you could simply adjust table structure as needed and then increase state version in State, so that it will automatically drop old state db and will reformat it with the new layout.

…ent error

efiop

Thanks @vyloy ! Please see a few more comments down below. 🙂

dvc/output/base.py

efiop · 2019-05-06T14:29:06Z

dvc/output/base.py

        )

    def changed_cache(self):
        if not self.use_cache or not self.checksum:
            return True

-        return self.cache.changed_cache(self.checksum)
+        return self.cache.changed_cache(self.checksum, self.checksum_type)


Same here, just pass checksum_info and let the remote deal with types.

efiop · 2019-05-06T14:33:28Z

dvc/remote/base.py

@@ -153,6 +153,12 @@ def group(self, name):
    def cache(self):
        return getattr(self.repo.cache, self.scheme)

+    def get_checksum_type_list(self):


Let's define something like
DEFAULT_CHECKSUMS = ["md5"]
for each remote instead of PARAM_CHECKSUM and then use it if checksums are not specified in the configs. Then make something like

@property def checksums(self): return self._checksums @property def checksum(self): return self._checksums[0]

and use that, so you don't have to define your getters for each remote.

Tried to define DEFAULT_CHECKSUM_TYPES but some tests like TestCacheExists are setting PARAM_CHECKSUM after init

Which is against from what you said, don't know how it got accepted in the code base 🙂

Overwriting constants is not nice.

Well, yeah, you'll have to adjust those to do things things properly.

It is acceptable for tests and unacceptable to overwrite constants in the core code itself.

It is acceptable for tests and unacceptable to overwrite constants in the core code itself.

IMHO, it is a bad decision.
Unacceptable things should be keeping consistent for both the core code and tests. They are under the same project, right? Find excuse to write unacceptable code in the project is not nice. 🙂
Otherwise, means the tests are using the core code in the unacceptable ways.

efiop · 2019-05-06T14:36:40Z

dvc/remote/base.py

@@ -180,7 +186,7 @@ def _collect_dir(self, path_info):
                dir_info.append(


Just make get_file_checksum return a dict {"checksum": "mychecksum"} and then use .update({self.PARAM_RELPATH: relpath}). Otherwise, you are splitting the knowtion of what checksum type get_file_checksum has returned and then assembling it all together, which is confusing.

Keep packing and unpacking checksum and checksum type as a dict is not efficient and is unnecessary.
Because generating new checksum will always use the preferred checksum type, which is always known here.

@property def checksums(self): return self._checksums @property def checksum(self): return self._checksums[0]

Because generating new checksum will always use the preferred checksum type, which is always known here.

It is not true though. Check out remote.changed_checksum, which needs to get_checksum of a particular type. Also, in "Remote.save_info" you are packing it into dict anyway. I'm just suggesting to keep it all together and not dispursing checksum notion everywhere. Check out your save_info:

return { checksum_type or self.checksum_type(): self.get_checksum( path_info, checksum_type ) }

with changes that I suggest it would be

return self.get_checksum(path_info)

🙂

It is not true though. Check out remote.changed_checksum, which needs to get_checksum of a particular type. Also, in "Remote.save_info" you are packing it into dict anyway. I'm just suggesting to keep it all together and not dispursing checksum notion everywhere. Check out your save_info:

That is why I said generating.
When you are passing a particular type to get_checksum, you knows what type of checksum it is returning, isn't it?
🙂 It is a legacy code, not my packing.

You do know if you specify that explicitly. But you are getting your checksum type from Remote while you are in the Output class, and then you are passing it back to Remote. Doesn't seem very optimal to me. You could return {"checksum": checksum} and let your remote decide by itself which default checksum type it wants to use.

Legacy code didn't deal with multiple checksums, and it is the point of this task to set it up so that it is able to do that properly.

You do know if you specify that explicitly. But you are getting your checksum type from Remote while you are in the Output class, and then you are passing it back to Remote. Doesn't seem very optimal to me. You could return {"checksum": checksum} and let your remote decide by itself which default checksum type it wants to use.

About But you are getting your checksum type from Remote while you are in the Output class, and then you are passing it back to Remote.
Not sure about where you are talking, but

dvc/dvc/output/base.py

Lines 143 to 148 in 00cfb04

@property

def checksum(self):

return self.info.get(self.remote.PARAM_CHECKSUM)

@property

shows the code was doing that since the beginning 🙂
And legacy code did deal with multiple checksums, because remotes don't use same checksums (types)

Here https://github.com/iterative/dvc/pull/1920/files#diff-9487de78a43eff53118660a866090b89R144 you are getting checksum type from remote and using it later.

The legacy code was mostly built around the assumption of 1 checksum per remote, hence PARAM_CHECKSUM. The point of the task that you are doing is to generalize it and do it properly.

The code was doing that since the beginning .🙂
Getting the checksum type from remote. self.info.get(self.remote.PARAM_CHECKSUM)
So, that does not change the fact that the legacy code still did deal with multiple checksums.

efiop · 2019-05-06T14:38:12Z

dvc/remote/base.py

@@ -195,8 +201,8 @@ def get_dir_checksum(self, path_info):
        if self.cache.changed_cache_file(checksum):
            self.cache.move(from_info, to_info)

-        self.state.save(path_info, checksum)
-        self.state.save(to_info, checksum)
+        self.state.save(path_info, checksum, self.get_prefer_checksum_type())


Does state really need to know checksum_type explicitly? Maybe we could just pass it a dict that looks something like {"md5": "mymd5"}? Relevant to the comment above.

self.state.save(path_info, checksum, self.get_prefer_checksum_type()) compare to self.state.save(path_info, {self.get_prefer_checksum_type(): checksum})

Which one do you like? Why?

Keep packing and unpacking checksum and checksum type as a dict is not efficient and is unnecessary.
And doing this means many refactoring with the legacy code.

But you are packing it into dict in state.save() anyway 🙂 And then you are manually updating particular keys. So why not just .update() already existing record with this new one? 🙂

Packing them into dict later is better, because it will save you an unpacking when you need to in the future. 🙂
Or in your design, you will pass multiple checksum types and checksums here?

Possibly, yes. The point is that you are not spreading checksum type notion across the code. State.save() doesn't need to know anything about checksum types, it just needs to save it.

dvc/remote/base.py

rename get_checksum_type_list to checksum_types

efiop · 2019-05-07T10:19:44Z

dvc/remote/base.py


        return checksum

-    def save_info(self, path_info):
+    def save_info(self, path_info, checksum_type=None):


I don't think save_info needs checksum_type parameter any longer, since it is not used anywhere ;)

efiop · 2019-05-09T11:14:34Z

@vyloy Looks like this #1920 (comment) comment got lost somewhere in your feed. Please take a look.

efiop · 2019-05-09T11:14:45Z

Oops, accidentally closed.

efiop · 2019-05-10T10:20:54Z

dvc/remote/hdfs.py

@@ -20,6 +21,8 @@ class RemoteHDFS(RemoteBase):
    REGEX = r"^hdfs://((?P<user>.*)@)?.*$"
    PARAM_CHECKSUM = "checksum"


So if you have SUPPORTED_CHECKSUM_TYPES down below, what do you need PARAM_CHECKSUM for ?

shcheklein · 2019-05-19T23:56:07Z

Closing in favor of #2022

efiop · 2019-10-16T18:51:36Z

Closing as stale. It is still linked to the original issue, so it won't be lost.

vyloy force-pushed the 1654 branch from 8c64bdb to 0b30f74 Compare April 25, 2019 07:28

efiop reviewed Apr 26, 2019

View reviewed changes

dvc/stage.py Outdated Show resolved Hide resolved

efiop reviewed Apr 26, 2019

View reviewed changes

vyloy changed the title ~~[WIP] config: Alterative hashings for data (#1676)~~ config: Alterative hashings for data (#1676) Apr 26, 2019

efiop suggested changes Apr 26, 2019

View reviewed changes

shcheklein reviewed Apr 27, 2019

View reviewed changes

dvc/config.py Outdated Show resolved Hide resolved

shcheklein reviewed Apr 27, 2019

View reviewed changes

dvc/remote/local.py Outdated Show resolved Hide resolved

shcheklein reviewed Apr 27, 2019

View reviewed changes

dvc/stage.py Outdated Show resolved Hide resolved

shcheklein reviewed Apr 27, 2019

View reviewed changes

shcheklein requested changes Apr 27, 2019

View reviewed changes

vyloy force-pushed the 1654 branch from 341071d to 97d777c Compare April 30, 2019 08:49

efiop suggested changes Apr 30, 2019

View reviewed changes

dvc/stage.py Outdated Show resolved Hide resolved

dvc/utils/hash/__init__.py Outdated Show resolved Hide resolved

dvc/stage.py Outdated Show resolved Hide resolved

efiop reviewed Apr 30, 2019

View reviewed changes

dvc/config.py Outdated Show resolved Hide resolved

config: Alterative hashings for data (iterative#1676)

e054d4e

vyloy added 9 commits May 5, 2019 10:57

remote: improvements after more tests

2a4bac3

remote: reformatted base.py

b230881

rename variable to fix python2.7 object does not support item assignm…

8a8b309

…ent error

rename mod hash to checksum

72ad029

refactor the code to deal with checksum type

81be420

rename legacy function about md5 to checksum

57be38d

collect schemas

46e3f29

increase state version

ad10e57

fix error after rebase

bb8d41a

vyloy force-pushed the 1654 branch from 902ccaf to bb8d41a Compare May 5, 2019 03:17

efiop suggested changes May 6, 2019

View reviewed changes

vyloy added 2 commits May 7, 2019 14:13

refactor remote.changed

20969a4

rename get_prefer_checksum_type to checksum_type

d034618

rename get_checksum_type_list to checksum_types

vyloy force-pushed the 1654 branch from 7183b68 to d034618 Compare May 7, 2019 06:13

efiop reviewed May 7, 2019

View reviewed changes

vyloy added 2 commits May 9, 2019 18:03

remote: initially add checksum types config ability for other remotes

9ea858a

remote: update save_info

f12cd8e

vyloy force-pushed the 1654 branch from 0580b7d to f12cd8e Compare May 9, 2019 10:03

efiop closed this May 9, 2019

efiop reopened this May 9, 2019

efiop reviewed May 10, 2019

View reviewed changes

shcheklein closed this May 19, 2019

shcheklein reopened this May 28, 2019

shcheklein changed the title ~~config: Alterative hashings for data (#1676)~~ [WIP] Alterative hashings for data (#1676) May 28, 2019

efiop closed this Oct 16, 2019

efiop mentioned this pull request Jan 7, 2020

Support hashings other than MD5 #3069

Open

		@@ -180,7 +186,7 @@ def _collect_dir(self, path_info):
		dir_info.append(


	@property
	def checksum(self):
	return self.info.get(self.remote.PARAM_CHECKSUM)

	@property

		@@ -20,6 +21,8 @@ class RemoteHDFS(RemoteBase):
		REGEX = r"^hdfs://((?P<user>.)@)?.$"
		PARAM_CHECKSUM = "checksum"

[WIP] Alterative hashings for data (#1676) #1920

[WIP] Alterative hashings for data (#1676) #1920

Conversation

vyloy commented Apr 24, 2019 • edited by shcheklein Loading

efiop commented Apr 24, 2019

efiop commented Apr 25, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyloy Apr 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyloy Apr 30, 2019 • edited Loading

Choose a reason for hiding this comment

vyloy Apr 30, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyloy May 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop May 1, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shcheklein left a comment

Choose a reason for hiding this comment

vyloy commented Apr 29, 2019 • edited Loading

efiop commented Apr 29, 2019

efiop commented Apr 29, 2019

shcheklein commented Apr 29, 2019

vyloy commented Apr 29, 2019

Suor commented Apr 30, 2019 • edited Loading

vyloy commented May 2, 2019

efiop commented May 2, 2019

efiop left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyloy May 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop May 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vyloy May 8, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop May 7, 2019 • edited Loading

Choose a reason for hiding this comment

efiop commented May 9, 2019

efiop commented May 9, 2019

efiop May 10, 2019 • edited Loading

Choose a reason for hiding this comment

shcheklein commented May 19, 2019

vyloy commented Apr 24, 2019 •

edited by shcheklein

Loading

vyloy Apr 30, 2019 •

edited

Loading

vyloy Apr 30, 2019 •

edited

Loading

vyloy Apr 30, 2019 •

edited

Loading

vyloy May 1, 2019 •

edited

Loading

efiop May 1, 2019 •

edited

Loading

vyloy commented Apr 29, 2019 •

edited

Loading

Suor commented Apr 30, 2019 •

edited

Loading

vyloy May 9, 2019 •

edited

Loading

efiop May 8, 2019 •

edited

Loading

vyloy May 8, 2019 •

edited

Loading

efiop May 7, 2019 •

edited

Loading

efiop May 10, 2019 •

edited

Loading