Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flash-481 Arrow encode #279

Merged
merged 150 commits into from
Oct 25, 2019
Merged
Changes from 1 commit
Commits
Show all changes
150 commits
Select commit Hold shift + click to select a range
aa92f4e
basic framework for coprocessor support in tiflash
windtalker Jul 30, 2019
4f37218
basic support for InterpreterDagRequestV2
windtalker Jul 30, 2019
85bfd5c
code refine
windtalker Jul 30, 2019
e1700c3
tipb submodule use tipb master branch
windtalker Jul 31, 2019
0f82665
rewrite build flow in InterpreterDagRequest
windtalker Jul 31, 2019
a7655bc
rename Dag to DAG
windtalker Jul 31, 2019
f516f00
Update tipb submodule
zanmato1984 Aug 1, 2019
3b520c9
basic support for selection/limit/topn executor in InterpreterDAGRequest
windtalker Aug 2, 2019
9591d26
Merge branch 'cop' of https://github.com/pingcap/tics into cop
windtalker Aug 2, 2019
ead9609
basic support for selection/limit/topn executor in InterpreterDAGRequ…
windtalker Aug 2, 2019
bed0bd4
merge pingcap/cop branch
windtalker Aug 2, 2019
526cad9
Code reorg
zanmato1984 Aug 4, 2019
be4d80c
Format
zanmato1984 Aug 4, 2019
64a45a9
merge pingcap/cop
windtalker Aug 5, 2019
a76fdb3
merge pingcap/cop
windtalker Aug 5, 2019
0cfe045
Refine code
zanmato1984 Aug 5, 2019
e9b216c
Merge branch 'cop' of https://github.com/pingcap/tics into cop
windtalker Aug 5, 2019
3617a87
basic support for dag agg executor
windtalker Aug 5, 2019
cb55df4
Code refine
zanmato1984 Aug 5, 2019
ed41c93
Merge master into cop
zanmato1984 Aug 5, 2019
08b7142
Refine code
zanmato1984 Aug 5, 2019
bc25942
Another way of getting codec flag
zanmato1984 Aug 5, 2019
059f267
fix cop test regression (#157)
windtalker Aug 6, 2019
e59e8f3
fix npe during dag execute (#160)
windtalker Aug 6, 2019
a618cb5
Add tipb cpp gen in build script
zanmato1984 Aug 6, 2019
4f797fe
Merge branch 'master' into cop
zanmato1984 Aug 6, 2019
bb51749
Fix build error and adjust some formats
zanmato1984 Aug 6, 2019
da1cb0e
Fix build error
zanmato1984 Aug 6, 2019
816ef4b
Fix build error
zanmato1984 Aug 6, 2019
f18fcdd
Update flash configs
zanmato1984 Aug 6, 2019
2ade1cb
Format
zanmato1984 Aug 6, 2019
3870d93
Merge branch 'master' into cop
zanmato1984 Aug 7, 2019
7cb9e71
throw exception when meet error duing cop request handling (#162)
windtalker Aug 7, 2019
5fe66ee
Merge branch 'master' into cop
zanmato1984 Aug 8, 2019
0174b7e
add DAGContext so InterpreterDAG can exchange information with DAGDri…
windtalker Aug 8, 2019
9a1dd23
columnref index is based on executor output schema (#167)
windtalker Aug 8, 2019
26e20d5
Move flash/cop/dag to individual library
zanmato1984 Aug 8, 2019
bf67d9d
Merge cop lib
zanmato1984 Aug 8, 2019
62ced38
DAG planner fix and mock dag request (#169)
zanmato1984 Aug 9, 2019
b346a24
Merge branch 'master' into cop
zanmato1984 Aug 9, 2019
57cd382
Fix DAG get and lock storage
zanmato1984 Aug 9, 2019
4a76e91
handle error in cop request (#171)
windtalker Aug 12, 2019
2d093a8
code refine && several minor bug fix (#174)
windtalker Aug 12, 2019
c8cd3d7
Fix region id in mock dag
zanmato1984 Aug 12, 2019
0492af6
support udf in (#175)
windtalker Aug 14, 2019
4a6bad8
Merge branch 'master' into cop
zanmato1984 Aug 14, 2019
8713ff2
1. fix decode literal expr error, 2. add all scalar function sig in s…
windtalker Aug 14, 2019
7759af1
Merge branch 'master' into cop
zanmato1984 Aug 15, 2019
b25d1cc
some bug fix (#179)
windtalker Aug 15, 2019
3d38b7b
Support all DAG operator types in mock SQL -> DAG parser (#176)
zanmato1984 Aug 15, 2019
cbcfdb0
filter column must be uint8 in tiflash (#180)
windtalker Aug 16, 2019
d87e2d5
1. fix encode null error, 2. fix empty field type generated by TiFlas…
windtalker Aug 16, 2019
17f7fcb
Merge branch 'master' into cop
zanmato1984 Aug 16, 2019
5853b91
check validation of dag exprs field type (#183)
windtalker Aug 19, 2019
0a6767a
Merge branch 'master' into cop
zanmato1984 Aug 19, 2019
d53ca34
Merge branch 'master' into cop
zanmato1984 Aug 20, 2019
5de0ec6
add more coprocessor mock tests (#185)
windtalker Aug 20, 2019
6196171
add some log about implicit cast (#188)
windtalker Aug 21, 2019
960cc56
Merge branch 'master' into cop
zanmato1984 Aug 24, 2019
08bacd7
Pass DAG tests after merging master (#199)
zanmato1984 Aug 24, 2019
e8b4198
Fix date/datetime/bit encode error (#200)
zanmato1984 Aug 26, 2019
61cdc8f
improve dag execution time collection (#202)
windtalker Aug 26, 2019
53dcd1f
Merge branch 'master' into cop
zanmato1984 Aug 27, 2019
10e3883
column id in table scan operator may be -1 (#205)
windtalker Aug 27, 2019
39d1994
quick fix for decimal encode (#210)
windtalker Aug 30, 2019
8a0fb66
support udf like with 3 arguments (#212)
windtalker Sep 2, 2019
ff9a1de
Flash-473 optimize date and datetime comparison (#221)
windtalker Sep 5, 2019
17aacde
Merge master
zanmato1984 Sep 5, 2019
6b14b38
FLASH-479 select from empty table throw error in tiflash (#223)
windtalker Sep 6, 2019
548e519
Update flash service port
zanmato1984 Sep 6, 2019
a1b8444
fix bug in DAGBlockOutputStream
windtalker Sep 10, 2019
fce3676
fix bug in DAGBlockOutputStream (#230)
windtalker Sep 10, 2019
a9f9b48
FLASH-475: Support BATCH COMMANDS in flash service (#232)
zanmato1984 Sep 12, 2019
bdc7d57
init change for array encode
windtalker Sep 12, 2019
516d340
merge pingcap/tics/cop
windtalker Sep 12, 2019
1ccfbd4
Merge branch 'master' into cop
zhexuany Sep 12, 2019
df07939
FLASH-483: Combine raft service and flash service (#235)
zanmato1984 Sep 16, 2019
99f26c0
Merge master
zanmato1984 Sep 16, 2019
0bb7991
Fix build error
zanmato1984 Sep 16, 2019
f41f853
Fix test regression
zanmato1984 Sep 16, 2019
259ec77
Fix null value bug in datum
zanmato1984 Sep 17, 2019
ef65514
Merge branch 'master' into cop
zanmato1984 Sep 17, 2019
708d52f
FLASH-490: Fix table scan with -1 column ID and no agg (#240)
zanmato1984 Sep 23, 2019
3656a95
Merge branch 'master' into cop
zanmato1984 Sep 23, 2019
a4c1074
throw error if the cop request is not based on full region scan (#247)
windtalker Sep 24, 2019
b57656c
Merge branch 'master' into cop
zanmato1984 Sep 25, 2019
3a43942
FLASH-437 Support time zone in coprocessor (#259)
windtalker Sep 27, 2019
01caa55
Merge branch 'master' into cop
zanmato1984 Sep 27, 2019
8d2576e
Address comment
zanmato1984 Sep 29, 2019
8ec5380
Merge branch 'cop' of https://github.com/pingcap/tics into array_encode
windtalker Sep 29, 2019
2e3b1c1
use the new date implementation
windtalker Sep 29, 2019
d33a278
FLASH-489 support key condition for coprocessor query (#261)
windtalker Sep 30, 2019
087faee
Merge branch 'master' into cop
zanmato1984 Sep 30, 2019
4aa2b58
only return execute summaies if requested (#264)
windtalker Sep 30, 2019
aed5e84
Merge branch 'cop' of https://github.com/pingcap/tics into array_encode
windtalker Oct 8, 2019
8663811
refine code
windtalker Oct 8, 2019
80f6f35
Refine service init (#265)
zanmato1984 Oct 8, 2019
0b737dc
fix bug
windtalker Oct 9, 2019
d3af009
fix bug
windtalker Oct 9, 2019
004f7c5
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 9, 2019
f255362
FLASH-554 cop check range should be based on region range (#270)
windtalker Oct 10, 2019
170f652
add ut for arrow encode
windtalker Oct 11, 2019
c53e456
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 11, 2019
7fc53ad
minor improve (#273)
windtalker Oct 11, 2019
22ad2d3
Merge branch 'master' into cop
zanmato1984 Oct 11, 2019
b01ccb3
update tipb
windtalker Oct 11, 2019
a1304ae
Fix mutex on timezone retrieval (#276)
ilovesoup2000 Oct 11, 2019
687dcbe
Fix race condition of batch command handling (#277)
zanmato1984 Oct 12, 2019
4dd5e1e
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 12, 2019
80c20b2
update tipb version
windtalker Oct 12, 2019
7c5bea6
set default record_per_chunk to 1024
windtalker Oct 13, 2019
939b8cf
address comment
windtalker Oct 14, 2019
d25dadc
address comments
windtalker Oct 14, 2019
512fa8e
refine code
windtalker Oct 14, 2019
ff9bf8f
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 14, 2019
a6f6dda
refine code
windtalker Oct 14, 2019
a943e8d
add mock_dag test
windtalker Oct 14, 2019
41272da
code refine
windtalker Oct 14, 2019
00dac75
code refine
windtalker Oct 14, 2019
4080fba
address comments
windtalker Oct 14, 2019
1188e69
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 14, 2019
d2890e3
Fix NULL order for dag (#281)
zanmato1984 Oct 14, 2019
bc075c5
refine get actions in DAGExpressionAnalyzer, fix bug in dbgFuncCoproc…
windtalker Oct 15, 2019
4dbff78
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 15, 2019
fbcbdc0
remove duplicate agg funcs (#283)
windtalker Oct 15, 2019
8f2bfaf
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 16, 2019
3716b98
refine code
windtalker Oct 16, 2019
fa42c69
remove useless code
windtalker Oct 16, 2019
7bbe8c0
address comments
windtalker Oct 16, 2019
31973bf
remove uselss include
windtalker Oct 16, 2019
d968c09
address comments
windtalker Oct 16, 2019
edf32d4
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 16, 2019
f1256bd
refine code
windtalker Oct 17, 2019
73befbd
address comments
windtalker Oct 17, 2019
3188c07
format code
windtalker Oct 17, 2019
87955d1
fix typo
windtalker Oct 17, 2019
4f58878
Update dbms/src/Flash/BatchCommandsHandler.cpp
zanmato1984 Oct 17, 2019
92c16c2
revert unnecessary changes
windtalker Oct 17, 2019
0f6f0a6
Merge branch 'cop' of https://github.com/pingcap/tics into arrow_encode
windtalker Oct 17, 2019
d550644
refine code
windtalker Oct 17, 2019
bac7951
fix build error
windtalker Oct 17, 2019
4a251b0
refine code
windtalker Oct 17, 2019
e8b92b4
Merge branch 'master' into cop
zanmato1984 Oct 17, 2019
48dd7bd
Merge master
zanmato1984 Oct 18, 2019
a8cba5f
Merge remote-tracking branch 'origin/cop' into arrow_encode_2
windtalker Oct 18, 2019
e3232af
Merge branch 'master' of https://github.com/pingcap/tics into arrow_e…
windtalker Oct 21, 2019
4d5e5d4
address comments
windtalker Oct 21, 2019
c7d8d4e
refine code
windtalker Oct 22, 2019
0b1ed77
address comments
windtalker Oct 25, 2019
683e7e0
Merge branch 'master' into arrow_encode
zanmato1984 Oct 25, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
code refine && several minor bug fix (#174)
* fix cop test regression

* address comments

* format code

* fix npe for dag execute

* format code

* address comment

* add some comments

* throw exception when meet error duing cop request handling

* address comments

* add error code

* throw exception when meet error duing cop request handling

* address comments

* add DAGContext so InterpreterDAG can exchange information with DAGDriver

* fix bug

* 1. refine code, 2. address comments

* update comments

* columnref index is based on executor output schema

* handle error in coprocessor request

* refine code

* use Clear to clear a protobuf message completely

* refine code

* code refine && several minor bug fix

* address comments

* address comments
windtalker authored and zanmato1984 committed Aug 12, 2019
commit 2d093a82d4c1e8c2f57cd59880a0afc32eca27ab
2 changes: 2 additions & 0 deletions dbms/src/Flash/Coprocessor/DAGBlockOutputStream.cpp
Original file line number Diff line number Diff line change
@@ -42,6 +42,7 @@ void DAGBlockOutputStream::writeSuffix()
if (current_chunk != nullptr && records_per_chunk > 0)
{
current_chunk->set_rows_data(current_ss.str());
dag_response.add_output_counts(records_per_chunk);
}
}

@@ -62,6 +63,7 @@ void DAGBlockOutputStream::write(const Block & block)
{
// set the current ss to current chunk
current_chunk->set_rows_data(current_ss.str());
dag_response.add_output_counts(current_records_num);
}
current_chunk = dag_response.add_chunks();
current_ss.str("");
2 changes: 1 addition & 1 deletion dbms/src/Flash/Coprocessor/DAGDriver.cpp
Original file line number Diff line number Diff line change
@@ -78,7 +78,7 @@ try
{
if (auto * p_stream = dynamic_cast<IProfilingBlockInputStream *>(streamPtr.get()))
{
time_processed_ns += p_stream->getProfileInfo().total_stopwatch.elapsed();
time_processed_ns = std::max(time_processed_ns, p_stream->getProfileInfo().total_stopwatch.elapsed());
num_produced_rows += p_stream->getProfileInfo().rows;
num_iterations += p_stream->getProfileInfo().blocks;
}
13 changes: 11 additions & 2 deletions dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.cpp
Original file line number Diff line number Diff line change
@@ -143,6 +143,15 @@ void DAGExpressionAnalyzer::appendOrderBy(ExpressionActionsChain & chain, const

const NamesAndTypesList & DAGExpressionAnalyzer::getCurrentInputColumns() { return after_agg ? aggregated_columns : source_columns; }

void DAGExpressionAnalyzer::appendFinalProject(ExpressionActionsChain & chain, const NamesWithAliases & final_project)
{
initChain(chain, getCurrentInputColumns());
for (auto name : final_project)
{
chain.steps.back().required_output.push_back(name.first);
}
}

void DAGExpressionAnalyzer::appendAggSelect(ExpressionActionsChain & chain, const tipb::Aggregation & aggregation)
{
initChain(chain, getCurrentInputColumns());
@@ -199,11 +208,11 @@ void DAGExpressionAnalyzer::appendAggSelect(ExpressionActionsChain & chain, cons

String DAGExpressionAnalyzer::appendCastIfNeeded(const tipb::Expr & expr, ExpressionActionsPtr & actions, const String & expr_name)
{
if (!expr.has_field_type())
if (!expr.has_field_type() && context.getSettingsRef().dag_expr_field_type_strict_check)
{
throw Exception("Expression without field type", ErrorCodes::COP_BAD_DAG_REQUEST);
}
if (isFunctionExpr(expr))
if (expr.has_field_type() && isFunctionExpr(expr))
{
DataTypePtr expected_type = getDataTypeByFieldType(expr.field_type());
DataTypePtr actual_type = actions->getSampleBlock().getByName(expr_name).type;
1 change: 1 addition & 0 deletions dbms/src/Flash/Coprocessor/DAGExpressionAnalyzer.h
Original file line number Diff line number Diff line change
@@ -44,6 +44,7 @@ class DAGExpressionAnalyzer : private boost::noncopyable
chain.steps.emplace_back(std::make_shared<ExpressionActions>(columns, settings));
}
}
void appendFinalProject(ExpressionActionsChain & chain, const NamesWithAliases & final_project);
String getActions(const tipb::Expr & expr, ExpressionActionsPtr & actions);
const NamesAndTypesList & getCurrentInputColumns();
};
4 changes: 2 additions & 2 deletions dbms/src/Flash/Coprocessor/DAGQuerySource.cpp
Original file line number Diff line number Diff line change
@@ -90,7 +90,7 @@ bool fillExecutorOutputFieldTypes(const tipb::Executor & executor, std::vector<t
switch (executor.tp())
{
case tipb::ExecType::TypeTableScan:
for (auto ci : executor.tbl_scan().columns())
for (auto & ci : executor.tbl_scan().columns())
{
field_type.set_tp(ci.tp());
field_type.set_flag(ci.flag());
@@ -136,7 +136,7 @@ std::vector<tipb::FieldType> DAGQuerySource::getResultFieldTypes() const
throw Exception("Do not found result field type for current dag request", ErrorCodes::COP_BAD_DAG_REQUEST);
}
// tispark assumes that if there is a agg, the output offset is
// ignored and the request out put is the same as the agg's output.
// ignored and the request output is the same as the agg's output.
// todo should always use output offset to re-construct the output field types
if (hasAggregation())
{
11 changes: 2 additions & 9 deletions dbms/src/Flash/Coprocessor/InterpreterDAG.cpp
Original file line number Diff line number Diff line change
@@ -208,15 +208,8 @@ InterpreterDAG::AnalysisResult InterpreterDAG::analyzeExpressions()
analyzer.appendOrderBy(chain, dag.getTopN(), res.order_column_names);
}
// Append final project results if needed.
// TODO: Refine this logic by an `analyzer.appendFinalProject()`-like call.
if (dag.hasSelection() || dag.hasAggregation() || dag.hasTopN())
{
for (auto & name : final_project)
{
chain.steps.back().required_output.push_back(name.first);
}
res.before_order_and_select = chain.getLastActions();
}
analyzer.appendFinalProject(chain, final_project);
res.before_order_and_select = chain.getLastActions();
chain.finalize();
chain.clear();
//todo need call prependProjectInput??
75 changes: 50 additions & 25 deletions dbms/src/Flash/Coprocessor/tests/cop_test.cpp
Original file line number Diff line number Diff line change
@@ -98,16 +98,11 @@ class FlashClient
};

using ClientPtr = std::shared_ptr<FlashClient>;
grpc::Status rpcTest()

void appendTS(tipb::DAGRequest & dag_request, size_t & result_field_num)
{
ChannelPtr cp = grpc::CreateChannel("localhost:9093", grpc::InsecureChannelCredentials());
ClientPtr clientPtr = std::make_shared<FlashClient>(cp);
size_t result_field_num = 0;
// construct a dag request
tipb::DAGRequest dagRequest;
dagRequest.set_start_ts(18446744073709551615uL);
// table scan: s,i
tipb::Executor * executor = dagRequest.add_executors();
tipb::Executor * executor = dag_request.add_executors();
executor->set_tp(tipb::ExecType::TypeTableScan);
tipb::TableScan * ts = executor->mutable_tbl_scan();
ts->set_table_id(44);
@@ -119,13 +114,16 @@ grpc::Status rpcTest()
ci->set_column_id(2);
ci->set_tp(8);
ci->set_flag(0);
dagRequest.add_output_offsets(1);
dagRequest.add_output_offsets(0);
dagRequest.add_output_offsets(1);
dag_request.add_output_offsets(1);
dag_request.add_output_offsets(0);
dag_request.add_output_offsets(1);
result_field_num = 3;
}

void appendSelection(tipb::DAGRequest & dag_request)
{
// selection: less(i, 123)
executor = dagRequest.add_executors();
auto * executor = dag_request.add_executors();
executor->set_tp(tipb::ExecType::TypeSelection);
tipb::Selection * selection = executor->mutable_selection();
tipb::Expr * expr = selection->add_conditions();
@@ -150,16 +148,19 @@ grpc::Status rpcTest()
type = expr->mutable_field_type();
type->set_tp(1);
type->set_flag(1 << 5);
}

void appendAgg(tipb::DAGRequest & dag_request, size_t & result_field_num)
{
// agg: count(s) group by i;
executor = dagRequest.add_executors();
auto * executor = dag_request.add_executors();
executor->set_tp(tipb::ExecType::TypeAggregation);
auto agg = executor->mutable_aggregation();
auto agg_func = agg->add_agg_func();
agg_func->set_tp(tipb::ExprType::Count);
auto child = agg_func->add_children();
child->set_tp(tipb::ExprType::ColumnRef);
ss.str("");
std::stringstream ss;
DB::EncodeNumber<Int64, TiDB::CodecFlagInt>(0, ss);
child->set_val(ss.str());
auto f_type = agg_func->mutable_field_type();
@@ -174,32 +175,56 @@ grpc::Status rpcTest()
f_type->set_tp(8);
f_type->set_flag(1);
result_field_num = 2;
}

// topn
/*
executor = dagRequest.add_executors();
void appendTopN(tipb::DAGRequest & dag_request)
{
auto * executor = dag_request.add_executors();
executor->set_tp(tipb::ExecType::TypeTopN);
tipb::TopN * topN = executor->mutable_topn();
topN->set_limit(3);
tipb::ByItem * byItem = topN->add_order_by();
byItem->set_desc(false);
tipb::Expr * expr1 = byItem->mutable_expr();
expr1->set_tp(tipb::ExprType::ColumnRef);
ss.str("");
std::stringstream ss;
DB::EncodeNumber<Int64, TiDB::CodecFlagInt>(1, ss);
expr1->set_val(ss.str());
type = expr1->mutable_field_type();
auto * type = expr1->mutable_field_type();
type->set_tp(8);
type->set_tp(0);
*/
// limit
/*
executor = dagRequest.add_executors();
}

void appendLimit(tipb::DAGRequest & dag_request)
{
auto * executor = dag_request.add_executors();
executor->set_tp(tipb::ExecType::TypeLimit);
tipb::Limit *limit = executor->mutable_limit();
tipb::Limit * limit = executor->mutable_limit();
limit->set_limit(5);
*/
}

grpc::Status rpcTest()
{
ChannelPtr cp = grpc::CreateChannel("localhost:9093", grpc::InsecureChannelCredentials());
ClientPtr clientPtr = std::make_shared<FlashClient>(cp);
size_t result_field_num = 0;
bool has_selection = false;
bool has_agg = true;
bool has_topN = false;
bool has_limit = false;
// construct a dag request
tipb::DAGRequest dagRequest;
dagRequest.set_start_ts(18446744073709551615uL);

appendTS(dagRequest, result_field_num);
if (has_selection)
appendSelection(dagRequest);
if (has_agg)
appendAgg(dagRequest, result_field_num);
if (has_topN)
appendTopN(dagRequest);
if (has_limit)
appendLimit(dagRequest);

// construct a coprocessor request
coprocessor::Request request;
4 changes: 3 additions & 1 deletion dbms/src/Flash/FlashService.cpp
Original file line number Diff line number Diff line change
@@ -96,8 +96,10 @@ std::tuple<Context, ::grpc::Status> FlashService::createDBContext(grpc::ServerCo
{
context.setSetting("dag_records_per_chunk", dag_records_per_chunk_str);
}
std::string planner = getClientMetaVarWithDefault(grpc_context, "dag_planner", "sql");
std::string planner = getClientMetaVarWithDefault(grpc_context, "dag_planner", "optree");
context.setSetting("dag_planner", planner);
std::string expr_field_type_check = getClientMetaVarWithDefault(grpc_context, "dag_expr_field_type_strict_check", "1");
context.setSetting("dag_expr_field_type_strict_check", expr_field_type_check);

return std::make_tuple(context, ::grpc::Status::OK);
}
1 change: 1 addition & 0 deletions dbms/src/Interpreters/Settings.h
Original file line number Diff line number Diff line change
@@ -31,6 +31,7 @@ struct Settings
M(SettingUInt64, read_tso, DEFAULT_MAX_READ_TSO, "tmt read tso.") \
M(SettingInt64, dag_records_per_chunk, DEFAULT_DAG_RECORDS_PER_CHUNK, "default chunk size of a DAG response.") \
M(SettingString, dag_planner, "sql", "planner for DAG query, sql builds the SQL string, optree builds the internal operator(stream) tree.") \
M(SettingBool, dag_expr_field_type_strict_check, true, "when set to true, every expr in the dag request must provide field type, otherwise only the result expr will be checked.") \
M(SettingInt64, schema_version, DEFAULT_UNSPECIFIED_SCHEMA_VERSION, "tmt schema version.") \
M(SettingUInt64, min_compress_block_size, DEFAULT_MIN_COMPRESS_BLOCK_SIZE, "The actual size of the block to compress, if the uncompressed data less than max_compress_block_size is no less than this value and no less than the volume of data for one mark.") \
M(SettingUInt64, max_compress_block_size, DEFAULT_MAX_COMPRESS_BLOCK_SIZE, "The maximum size of blocks of uncompressed data before compressing for writing to a table.") \
9 changes: 9 additions & 0 deletions dbms/src/Storages/Transaction/Codec.h
Original file line number Diff line number Diff line change
@@ -324,6 +324,11 @@ inline void EncodeVarUInt(UInt64 num, std::stringstream & ss)
TiKV::writeVarUInt(num, ss);
}

inline void EncodeNull(std::stringstream &ss)
{
writeIntBinary(UInt8(TiDB::CodecFlagNil), ss);
}

inline void EncodeDecimal(const Decimal & dec, std::stringstream & ss)
{
writeIntBinary(UInt8(TiDB::CodecFlagDecimal), ss);
@@ -394,6 +399,10 @@ T getFieldValue(const Field & field)

inline void EncodeDatum(const Field & field, TiDB::CodecFlag flag, std::stringstream & ss)
{
if (field.isNull())
{
return EncodeNull(ss);
}
switch (flag)
{
case TiDB::CodecFlagDecimal: