-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[db voltdb] Add VoltDB client #1319
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks for the effort on merging this in.
it looks like the bin
directory is in your fork here:
https://github.com/srmadscience/YCSB/tree/db_voltdb/bin
don't forget to update both bindings.properties
and the mapping in ycsb
Sean,
Thank you for getting back to me so quickly!
1. I am meeting our VP of Eng this Sunday (she's visiting Dublin...) and
I'll ask if the *GPL issue is a problem for us - AFAIK it's not...
2. I will aim to have another commit done by COB next Wednesday. I'm away
until Tue ...
3. My goal is to get this across the line with the minimum of ping-ponging
back and forth...
Regards,
David Rolfe
*David Rolfe* | Director of Solutions Engineering (EMEA)
*e:* [email protected] <[email protected]>*m:* +353.86.173.6280
voltdb.com <https://www.facebook.com/VoltDB/> <https://twitter.com/VoltDB>
<https://www.linkedin.com/company/voltdb>
…On Wed, Jul 3, 2019 at 2:38 PM Sean Busbey ***@***.***> wrote:
***@***.**** requested changes on this pull request.
thanks for the effort on merging this in.
it looks like the bin directory is in your fork here:
https://github.com/srmadscience/YCSB/tree/db_voltdb/bin
don't forget to update both bindings.properties and the mapping in ycsb
------------------------------
In voltdb/README.md
<#1319 (comment)>
:
> @@ -0,0 +1,68 @@
+# YCSB for VoltDB
needs a license header
------------------------------
In voltdb/pom.xml
<#1319 (comment)>
:
> + <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ <scope>test</scope>
+ </dependency>
+<!-- https://mvnrepository.com/artifact/org.voltdb/voltdbclient -->
+<dependency>
+ <groupId>org.voltdb</groupId>
+ <artifactId>voltdbclient</artifactId>
+ <version>9.0</version>
+</dependency>
+<!-- https://mvnrepository.com/artifact/org.voltdb/voltdb -->
+<dependency>
+ <groupId>org.voltdb</groupId>
+ <artifactId>voltdb</artifactId>
just curious, why this dependency if we already have the volddbclient
dependency?
------------------------------
In voltdb/pom.xml
<#1319 (comment)>
:
> + <dependency>
+ <groupId>com.yahoo.ycsb</groupId>
+ <artifactId>core</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>junit</groupId>
+ <artifactId>junit</artifactId>
+ <version>4.12</version>
+ <scope>test</scope>
+ </dependency>
+<!-- https://mvnrepository.com/artifact/org.voltdb/voltdbclient -->
+<dependency>
+ <groupId>org.voltdb</groupId>
+ <artifactId>voltdbclient</artifactId>
voltdbclient appears to be AGPL licensed according to its maven central
pom:
https://search.maven.org/artifact/org.voltdb/voltdbclient/9.0/jar
I'll need to review our other dependencies; I don't think we've allowed
*GPL dependencies before.
------------------------------
In voltdb/src/main/java/com/yahoo/ycsb/db/VoltClient4.java
<#1319 (comment)>
:
> + * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+/*
+ * This client provideds a wrapper layer for running the Yahoo Cloud Serving
+ * Benchmark (YCSB) against VoltDB. This benchmark runs a synchronous client
+ * with a mix of the operations provided below. YCSB is open-source, and may
+ * be found at https://github.com/brianfrankcooper/YCSB. The YCSB jar must be
+ * in your classpath to compile this client.
+ */
+package com.yahoo.ycsb.db;
this should be in com.yahoo.ycsb.db.voltdb
------------------------------
In voltdb/src/main/java/com/yahoo/ycsb/db/VoltClient4.java
<#1319 (comment)>
:
> +import org.voltdb.client.Client;
+import org.voltdb.client.ClientResponse;
+import org.voltdb.client.ClientResponseWithPartitionKey;
+import org.voltdb.client.ProcCallException;
+import org.voltdb.sortedvolttable.VoltDBTableSortedMergeWrangler;
+import org.voltdb.ycsb.ConnectionHelper;
+import org.voltdb.ycsb.YCSBSchemaBuilder;
+
+import com.yahoo.ycsb.ByteArrayByteIterator;
+import com.yahoo.ycsb.ByteIterator;
+import com.yahoo.ycsb.DB;
+import com.yahoo.ycsb.DBException;
+import com.yahoo.ycsb.Status;
+
+/**
+ * @author srmadscience / VoltDB
please leave out author tags; we rely on git commit metadata for that kind
of tracking.
------------------------------
In voltdb/src/main/java/com/yahoo/ycsb/db/package-info.java
<#1319 (comment)>
:
> @@ -0,0 +1,7 @@
+/**
+ * VoltDB integration with YCSB.
+ *
+ * @author srmadscience / VoltDB
+ *
+ */
+package com.yahoo.ycsb.db;
shouldn't need this package-info file because everything should be in
com.yahoo.ycsb.db.voltdb
------------------------------
In
voltdb/src/main/java/org/voltdb/sortedvolttable/ClientResponseIsBadException.java
<#1319 (comment)>
:
> @@ -0,0 +1,31 @@
+package org.voltdb.sortedvolttable;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In
voltdb/src/main/java/org/voltdb/sortedvolttable/IncomingVoltTablesNeedToBeSortedException.java
<#1319 (comment)>
:
> @@ -0,0 +1,31 @@
+package org.voltdb.sortedvolttable;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In
voltdb/src/main/java/org/voltdb/sortedvolttable/NeedsToBeComparableException.java
<#1319 (comment)>
:
> @@ -0,0 +1,34 @@
+package org.voltdb.sortedvolttable;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In
voltdb/src/main/java/org/voltdb/sortedvolttable/VoltDBTableSortedMergeWrangler.java
<#1319 (comment)>
:
> @@ -0,0 +1,207 @@
+package org.voltdb.sortedvolttable;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdb/sortedvolttable/package-info.java
<#1319 (comment)>
:
> @@ -0,0 +1,14 @@
+/**
+ *
+ * VoltDBTableSortedMergeWrangler allows you to merge an array of VoltTable
+ * provided by callAllPartitionProcedure.
+ *
+ * The intended use case is for when you need to issue a multi partition query
+ * but would prefer not to, as you don't need perfect read consistency and would
+ * rather get the individual VoltDB partitions to issue the query independently
+ * and then somehow merge the results.
+ *
+ * @author srmadscience / VoltDB
+ *
+ */
+package org.voltdb.sortedvolttable;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdb/ycsb/ConnectionHelper.java
<#1319 (comment)>
:
> + * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+/*
+ * Manage a mapping of one VoltDB client to every 50 YCSB client threads.
+ */
+package org.voltdb.ycsb;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdb/ycsb/YCSBSchemaBuilder.java
<#1319 (comment)>
:
> + * Copyright (c) 2015-2019 YCSB contributors. All rights reserved.
+ * <p>
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License. You
+ * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+package org.voltdb.ycsb;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdb/ycsb/package-info.java
<#1319 (comment)>
:
> @@ -0,0 +1,7 @@
+/**
+ * VoltDB integration with YCSB.
+ *
+ * @author srmadscience / VoltDB
+ *
+ */
+package org.voltdb.ycsb;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdbycsb/procs/ByteWrapper.java
<#1319 (comment)>
:
> + * <p>
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License. You
+ * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+package org.voltdbycsb.procs;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdbycsb/procs/Put.java
<#1319 (comment)>
:
> + * <p>
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License. You
+ * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+package org.voltdbycsb.procs;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdbycsb/procs/Scan.java
<#1319 (comment)>
:
> + * <p>
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License. You
+ * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+package org.voltdbycsb.procs;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdbycsb/procs/ScanAll.java
<#1319 (comment)>
:
> + * <p>
+ * Licensed under the Apache License, Version 2.0 (the "License"); you
+ * may not use this file except in compliance with the License. You
+ * may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+package org.voltdbycsb.procs;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
------------------------------
In voltdb/src/main/java/org/voltdbycsb/procs/package-info.java
<#1319 (comment)>
:
> + *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ * implied. See the License for the specific language governing
+ * permissions and limitations under the License. See accompanying
+ * LICENSE file.
+ */
+
+/**
+ * VoltDB org.voltdbycsb.procs for Put and Scan. Other org.voltdbycsb.procs
+ * are defined using DDL in ycsb_ddl.sql
+ */
+package org.voltdbycsb.procs;
this should be in com.yahoo.ycsb.db.voltdb or a package under that
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1319?email_source=notifications&email_token=AFOIDYUDRFCNIPSUUEIOJJ3P5STXBA5CNFSM4H4ZLZE2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOB5MO43A#pullrequestreview-257486444>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFOIDYXSJ23OTT2LRZ4IXCDP5STXBANCNFSM4H4ZLZEQ>
.
|
Sean,
Regards, David Rolfe |
Hi David! Sorry for the lag on chasing this down.
I think that varies by project. Personally I'm fine either way. Resolving them myself helps me get context when picking a review back up, but I am also fine just reviewing the then-current code without those markers.
The issue isn't your VoltDB specific YCSB code (though since our project license is ALv2, we would require contributions to be under that license). The issue is that adding the Volt DB client libraries as a dependency would bring them in as a part of our binary packaging and add them as a transitive dependency should others rely on us. Having those libraries be AGPL presents a problem since it is more restrictive then the license we have for everything else. For example, any ASF projects that rely on us would have to take pains to make sure they don't get that part of the code base. As I mentioned though, I haven't done a dependency review in some time to ensure we don't have other *GPL dependencies sitting around. I will try to find time to do that. Is there any chance of VoltDB sticking with whatever licensing for the server code and releasing client specific libraries under ALv2 instead of AGPL? IIRC there are some other data stores that did similar, either with GPL or closed source servers that are accessed via a client that's under a permissive open source license. |
Sean,
I'll go ask the appropriate people. If you get a chance can you eyeball my
other changes?
Regards,
David Rolfe
*David Rolfe* | Director of Solutions Engineering (EMEA)
*e:* [email protected] <[email protected]>*m:* +353.86.173.6280
voltdb.com <https://www.facebook.com/VoltDB/> <https://twitter.com/VoltDB>
<https://www.linkedin.com/company/voltdb>
…On Tue, Aug 20, 2019 at 3:39 PM Sean Busbey ***@***.***> wrote:
Hi David!
Sorry for the lag on chasing this down.
One item of GitHub etiquette: Can I confirm it's your job to 'Resolve'
comments, not mine?
I think that varies by project. Personally I'm fine either way. Resolving
them myself helps me get context when picking a review back up, but I am
also fine just reviewing the then-current code without those markers.
I talked to our VP Eng and we have no issues using your licensing for the
VoltDB specific YCSB code.
The issue isn't your VoltDB specific YCSB code (though since our project
license is ALv2, we would require contributions to be under that license).
The issue is that adding the Volt DB client libraries as a dependency would
bring them in as a part of our binary packaging and add them as a
transitive dependency should others rely on us. Having those libraries be
AGPL presents a problem since it is more restrictive then the license we
have for everything else. For example, any ASF projects that rely on us
would have to take pains to make sure they don't get that part of the code
base.
As I mentioned though, I haven't done a dependency review in some time to
ensure we don't have other *GPL dependencies sitting around. I will try to
find time to do that.
Is there any chance of VoltDB sticking with whatever licensing for the
server code and releasing client specific libraries under ALv2 instead of
AGPL? IIRC there are some other data stores that did similar, either with
GPL or closed source servers that are accessed via a client that's under a
permissive open source license.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#1319?email_source=notifications&email_token=AFOIDYXCREANV7KSBGWN56TQFP62NA5CNFSM4H4ZLZE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4WQVKQ#issuecomment-523045546>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFOIDYXR2H25FFFQ2MLNHTDQFP62NANCNFSM4H4ZLZEQ>
.
|
voltdb/src/main/java/com/yahoo/ycsb/db/voltdb/ConnectionHelper.java
Outdated
Show resolved
Hide resolved
voltdb/src/main/java/com/yahoo/ycsb/db/voltdb/ConnectionHelper.java
Outdated
Show resolved
Hide resolved
voltdb/src/main/java/org/voltdb/sortedvolttable/ClientResponseIsBadException.java
Outdated
Show resolved
Hide resolved
voltdb/src/test/java/com/yahoo/ycsb/db/voltdb/test/VoltDBClientTest.java
Show resolved
Hide resolved
Sean, Spoke to our VP/Eng. We're looking at what's involved in changing to Apache for the client. In the mean time I'll work on the other items... DR |
Sean, We can fix it so the client uses the MIT licence. Will that work for you guys? David Rolfe |
Yeah, that'd be great. |
As far as I can tell We've fixed all the issues that have been raised so far. Can someone review? Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we're getting close!
I also plan to download the client jar to make sure the license metadata in it matches the pom.
Do y'all have plans to call out the license change on the client jar anywhere?
voltdb/src/main/java/com/yahoo/ycsb/db/voltdb/ConnectionHelper.java
Outdated
Show resolved
Hide resolved
voltdb/src/main/java/com/yahoo/ycsb/db/voltdb/ConnectionHelper.java
Outdated
Show resolved
Hide resolved
voltdb/src/main/java/com/yahoo/ycsb/db/voltdb/YCSBSchemaBuilder.java
Outdated
Show resolved
Hide resolved
voltdb/src/test/java/com/yahoo/ycsb/db/voltdb/test/VoltDBClientTest.java
Outdated
Show resolved
Hide resolved
voltdb/src/test/java/com/yahoo/ycsb/db/voltdb/test/VoltDBClientTest.java
Show resolved
Hide resolved
voltdb/src/test/java/com/yahoo/ycsb/db/voltdb/test/VoltDBClientTest.java
Outdated
Show resolved
Hide resolved
I've hit an issue I need some advice on. If I change the VoltDB lib to runtime or test dependency the build fails because it sees the stored procedure classes, tries to compile them, and fails because the server libs aren't in the path. My thinking is to dynamically write the stored procedure classes - i.e. they would cease to exist as Java classes and turn into Strings that are used to compile classes when we have determined we are connected to a real, live copy of VoltDB. Is this the 'least worst' solution to this problem? |
Try |
'provided' worked. |
As far as I can tell everything raised on this pass has been done... |
pom.xml
Outdated
@@ -110,6 +110,7 @@ LICENSE file. | |||
<thrift.version>0.8.0</thrift.version> | |||
<voldemort.version>0.81</voldemort.version> | |||
<tablestore.version>4.8.0</tablestore.version> | |||
<voltdb.version>9.1</voltdb.version> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought it was specifically version 9.1.1 we needed? Also it doesn't look like the voltdb-binding pom is using this currently?
|
||
int ratelimit = strLimit != null ? Integer.parseInt(strLimit) : Integer.MAX_VALUE; | ||
try { | ||
mclient = ConnectionHelper.createConnection(Thread.currentThread().getId(), servers, user, password, ratelimit); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if clientId isn't needed anymore, could it get removed? this call here is still not guaranteed to pass a unique long for each DB instance, since a thread could get reused.
private static boolean haveDb = false; | ||
|
||
@BeforeClass | ||
public static void setup() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you should be able to do a single Assume.assumeTrue(haveDb);
at the end of this setup
method instead of doing one on each test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried this and got:
Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 0.439 sec <<< FAILURE! - in com.yahoo.ycsb.db.voltdb.test.VoltDBClientTest
com.yahoo.ycsb.db.voltdb.test.VoltDBClientTest Time elapsed: 0.439 sec <<< ERROR!
org.junit.AssumptionViolatedException: got: , expected: is
at org.junit.Assume.assumeThat(Assume.java:95)
at org.junit.Assume.assumeTrue(Assume.java:41)
at com.yahoo.ycsb.db.voltdb.test.VoltDBClientTest.setup(VoltDBClientTest.java:100)com.yahoo.ycsb.db.voltdb.test.VoltDBClientTest Time elapsed: 0.439 sec <<< ERROR!
org.junit.AssumptionViolatedException: got: , expected: is
at org.junit.Assume.assumeThat(Assume.java:95)
at org.junit.Assume.assumeTrue(Assume.java:41)
at com.yahoo.ycsb.db.voltdb.test.VoltDBClientTest.teardown(VoltDBClientTest.java:124)
I've put it back the way it was - do you know how to get it to work with a single 'Assume' or shall we leave it as is?
Other than the "Assume.assumeTrue(haveDb)" niggle everything appear to be OK now... |
Yep, I agree. Thanks for keeping at this David! |
VoltDB has had a YCSB implementation for a few years now - see https://github.com/VoltDB/voltdb/tree/master/tests/test_apps/ycsb . While we supported the benchmark we'd never get round to formally integrating it with the 'official' YCSB repo. This pull request will hopefully address that.
In addition to moving the code around and fixing it so the init() method creates the schema if needed I've also made what I believe to the required changes to two pom.xml files. What I haven't been able to do is figure out how this line - which is needed - AFAIK - ends up in 'bindings.properties', as the 'bin' directory isn't in my fork:
voltdb:com.yahoo.ycsb.db.VoltClient4
I have built this on ubuntu on AWS using the Oracle JDK, and have created the schema and run workloads a - e successfully.
I am based in Dublin, Ireland and can be reached at drolfe at voltdb.com.
Regards,
DR