Skip to content

use Rcpp to convert R Dates to Python #643

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 19, 2019
Merged

use Rcpp to convert R Dates to Python #643

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d44ed21
use Rcpp to convert R Dates to Python
Nov 13, 2019
58287b1
remove datetime.h and use datetime module instead
Nov 15, 2019
a20d255
handle scalars and vectors
Nov 15, 2019
73a1014
don't wrap in PyObjectPtr
Nov 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions R/RcppExports.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,10 @@ r_convert_dataframe <- function(dataframe, convert) {
.Call(`_reticulate_r_convert_dataframe`, dataframe, convert)
}

r_convert_date <- function(date_vector, convert) {
.Call(`_reticulate_r_convert_date`, date_vector, convert)
}

readline <- function(prompt) {
.Call(`_reticulate_readline`, prompt)
}
Expand Down
15 changes: 1 addition & 14 deletions R/conversion.R
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -159,20 +159,7 @@ py_to_r.datetime.datetime <- function(x) {

#' @export
r_to_py.Date <- function(x, convert = FALSE) {

datetime <- import("datetime", convert = FALSE)
items <- lapply(x, function(item) {
iso <- strsplit(format(item), "-", fixed = TRUE)[[1]]
year <- as.integer(iso[[1]])
month <- as.integer(iso[[2]])
day <- as.integer(iso[[3]])
datetime$date(year, month, day)
})

if (length(items) == 1)
items[[1]]
else
r_to_py_impl(items, convert)
r_convert_date(x, convert)
Copy link
Collaborator

@kevinushey kevinushey Nov 13, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may need to handle length-one lists still (to ensure they're returned as scalars)

Copy link
Author

@skeydan skeydan Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure exactly what you mean? I was trying to find what this

reticulate/R/conversion.R

Line 172 in 64f685a

if (length(items) == 1)

was for but didn't manage to get into the else branch...

Copy link
Collaborator

@kevinushey kevinushey Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g.

> library(reticulate)
> r_to_py(Sys.Date())
2019-11-15
> r_to_py(c(Sys.Date(), Sys.Date()))
[datetime.date(2019, 11, 15), datetime.date(2019, 11, 15)]

That is, we make a Python list for R vectors of length > 1, and a 'scalar' for R vectors of length 1. Is this behavior still preserved in this PR?

Copy link
Author

@skeydan skeydan Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, got it, thanks! Does this look better now?

}

#' @export
Expand Down
13 changes: 13 additions & 0 deletions src/RcppExports.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -531,6 +531,18 @@ BEGIN_RCPP
return rcpp_result_gen;
END_RCPP
}
// r_convert_date
PyObjectRef r_convert_date(DateVector date_vector, bool convert);
RcppExport SEXP _reticulate_r_convert_date(SEXP date_vectorSEXP, SEXP convertSEXP) {
BEGIN_RCPP
Rcpp::RObject rcpp_result_gen;
Rcpp::RNGScope rcpp_rngScope_gen;
Rcpp::traits::input_parameter< DateVector >::type date_vector(date_vectorSEXP);
Rcpp::traits::input_parameter< bool >::type convert(convertSEXP);
rcpp_result_gen = Rcpp::wrap(r_convert_date(date_vector, convert));
return rcpp_result_gen;
END_RCPP
}
// readline
SEXP readline(const std::string& prompt);
RcppExport SEXP _reticulate_readline(SEXP promptSEXP) {
Expand Down Expand Up @@ -590,6 +602,7 @@ static const R_CallMethodDef CallEntries[] = {
{"_reticulate_py_convert_pandas_series", (DL_FUNC) &_reticulate_py_convert_pandas_series, 1},
{"_reticulate_py_convert_pandas_df", (DL_FUNC) &_reticulate_py_convert_pandas_df, 1},
{"_reticulate_r_convert_dataframe", (DL_FUNC) &_reticulate_r_convert_dataframe, 2},
{"_reticulate_r_convert_date", (DL_FUNC) &_reticulate_r_convert_date, 2},
{"_reticulate_readline", (DL_FUNC) &_reticulate_readline, 1},
{NULL, NULL, 0}
};
Expand Down
1 change: 1 addition & 0 deletions src/libpython.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,6 +244,7 @@ bool LibPython::loadSymbols(bool python3, std::string* pError)
LOAD_PYTHON_SYMBOL(PyObject_CallMethod)
LOAD_PYTHON_SYMBOL(PySequence_GetItem)
LOAD_PYTHON_SYMBOL(PyObject_IsTrue)
LOAD_PYTHON_SYMBOL(PyCapsule_Import)

// PyUnicode_AsEncodedString may have several different names depending on the Python
// version and the UCS build type
Expand Down
4 changes: 4 additions & 0 deletions src/libpython.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,8 @@ typedef struct PyCompilerFlags{
int cf_feature_version;
} PyCompilerFlags;

typedef Py_ssize_t Py_hash_t;

LIBPYTHON_EXTERN PyTypeObject* PyFunction_Type;
LIBPYTHON_EXTERN PyTypeObject* PyModule_Type;
LIBPYTHON_EXTERN PyTypeObject* PyType_Type;
Expand Down Expand Up @@ -317,6 +319,8 @@ LIBPYTHON_EXTERN int (*PyObject_IsTrue)(PyObject *o);

LIBPYTHON_EXTERN PyObject* (*Py_CompileStringExFlags)(const char *str, const char *filename, int start, PyCompilerFlags *flags, int optimize);

LIBPYTHON_EXTERN void* (*PyCapsule_Import)(const char *name, int no_block);

#define PyObject_TypeCheck(o, tp) ((PyTypeObject*)Py_TYPE(o) == (tp)) || PyType_IsSubtype((PyTypeObject*)Py_TYPE(o), (tp))

#define PyType_Check(o) PyObject_TypeCheck(o, PyType_Type)
Expand Down
43 changes: 43 additions & 0 deletions src/python.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1702,6 +1702,7 @@ void py_initialize(const std::string& python,

// poll for events while executing python code
event_loop::initialize();

}

// [[Rcpp::export]]
Expand Down Expand Up @@ -2415,3 +2416,45 @@ PyObjectRef r_convert_dataframe(RObject dataframe, bool convert) {
return py_ref(dict.detach(), convert);

}

// [[Rcpp::export]]
PyObjectRef r_convert_date(DateVector date_vector, bool convert) {

PyObjectPtr datetime(PyImport_ImportModule("datetime"));

// scalar
if (date_vector.size() == 1) {

Date date = date_vector[0];
PyObject* py_date(PyObject_CallMethod(
datetime, "date", "iii",
static_cast<int>(date.getYear()),
static_cast<int>(date.getMonth()),
static_cast<int>(date.getDay())));
if (py_date == NULL) {
stop(py_fetch_error());
}
return py_ref(py_date, convert);
Copy link
Collaborator

@kevinushey kevinushey Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need py_date.detach()?

Copy link
Author

@skeydan skeydan Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed this one to PyObject* now too


// vector
} else {

PyObjectPtr list(PyList_New(date_vector.size()));
for (int i = 0; i < date_vector.size(); ++i) {
Date date = date_vector[i];
PyObject* py_date(PyObject_CallMethod(
datetime, "date", "iii",
static_cast<int>(date.getYear()),
static_cast<int>(date.getMonth()),
static_cast<int>(date.getDay())));
if (py_date == NULL) {
stop(py_fetch_error());
}
int res = PyList_SetItem(list, i, py_date);
Copy link
Collaborator

@kevinushey kevinushey Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PyList_SetItem "steals" a reference, and so the associated py_date item should either not be wrapped in PyObjectPtr, or explicitly detached. See for example:

reticulate/src/python.cpp

Lines 1309 to 1313 in 64f685a

PyObject* item = r_to_py(RObject(VECTOR_ELT(sexp, i)), convert);
// NOTE: reference to added value is "stolen" by the list
int res = PyList_SetItem(list, i, item);
if (res != 0)
stop(py_fetch_error());

Copy link
Collaborator

@kevinushey kevinushey Nov 15, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This is one of the frustrating bits of the Python API: most functions return new references, but some 'borrow' or 'steal' a reference and so have implications on how reference counting is performed)

Copy link
Author

@skeydan skeydan Nov 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh sorry, I should have connected the dots ... (had read about e.g. PyList_SetItem() and also seen PyObjectPtr* in the piece of code I was taking as an example...)

if (res != 0)
stop(py_fetch_error());
}
return py_ref(list.detach(), convert);
}

}