Refine the documentation of im2rec #12606

stu1130 · 2018-09-19T23:01:44Z

Description

Currently, we have two im2rec tools. One is python, the other one is C++. There are slightly different in terms of functionality. It helps to solve the #11884 as well.

change the data format on example
add some notes to tell users the difference and add corresponding to each other.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Comments

@aaronmarkham
Please let me know how can I make it less confused.

stu1130 · 2018-09-19T23:05:48Z

@mxnet-label-bot[pr-awaiting-review]

lupesko

Looks good, just one clarification.

lupesko · 2018-09-20T00:33:13Z

docs/faq/recordio.md

-972457  633     n07723039_1627.JPEG
-7534    11      n01630670_4486.JPEG
-1191261 249     n12407079_5106.JPEG
+95099  464.000000     n04467665_17283.JPEG


Is the decimal point really required? this is a bit weird.

If the data.lst file is generated by im2rec.py instead of doing it manually, the label will have those decimal point. I think it would be less confused for users?
And the reason why it uses floating point is that the label value could be generated by the regression, e.g. 68.6 kg for a human body weight.

sandeep-krishnamurthy

LGTM.
However, we need to revisit why do we need 2 versions and if required, why is there a discrepency in functionality.

aaronmarkham

I realize this was already merged, but I have suggestions nonetheless.

aaronmarkham · 2018-09-20T01:26:24Z

docs/faq/recordio.md

@@ -6,35 +6,39 @@ RecordIO implements a file format for a sequence of records. We recommend storin
 * Packing data together allows continuous reading on the disk.
 * RecordIO has a simple way to partition, simplifying distributed setting. We provide an example later.

-We provide the [im2rec tool](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc) so you can create an Image RecordIO dataset by yourself. The following walkthrough shows you how.
+We provide the [im2rec tool](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc) so you can create an Image RecordIO dataset by yourself. The following walkthrough shows you how. Note that there is python version of [im2rec tool](https://github.com/apache/incubator-mxnet/blob/master/tools/im2rec.py) and [example](https://mxnet.incubator.apache.org/tutorials/basic/data.html) using real-world data.


We provide two tools for creating a RecordIO dataset.

im2rec.cc - implements the tool using the C++ API.

im2rec.py - implements the tool using the Python API.

Both provide the same output: a RecordIO dataset.
(Then take this mention and add it later for "Next Steps". I don't think you want them leaving this FAQ/tutorial quite yet.)
You may want to also review the example using real-world data with im2rec.py.

aaronmarkham · 2018-09-20T16:41:02Z

docs/faq/recordio.md

 Download the data. You don't need to resize the images manually. You can use ```im2rec``` to resize them automatically. For details, see the "Extension: Using Multiple Labels for a Single Image," later in this topic.

 ### Step 1. Make an Image List File
+
+* Note that the im2rec.py provide a param `--list` to generate the list for you but im2rec.cc doesn't support it.


aaronmarkham · 2018-09-20T16:42:18Z

docs/tutorials/basic/data.md

@@ -315,6 +315,8 @@ print(mx.recordio.unpack_img(s))
 You can also convert raw images into *RecordIO* format using the ``im2rec.py`` utility script that is provided in the MXNet [src/tools](https://github.com/dmlc/mxnet/tree/master/tools) folder.


Update the repo link

aaronmarkham · 2018-09-20T16:44:31Z

docs/tutorials/basic/data.md

@@ -315,6 +315,8 @@ print(mx.recordio.unpack_img(s))
 You can also convert raw images into *RecordIO* format using the ``im2rec.py`` utility script that is provided in the MXNet [src/tools](https://github.com/dmlc/mxnet/tree/master/tools) folder.
 An example of how to use the script for converting to *RecordIO* format is shown in the `Image IO` section below.

+* Note that there is a C++ version of [im2rec](https://github.com/dmlc/mxnet/blob/master/tools/im2rec.cc), please refer to [here](https://mxnet.incubator.apache.org/faq/recordio.html) for more information.


Please don't link only "here". Provide the full description of what the link is going to.
Note that there is a C++ API implementation of im2rec. Refer to the RecordIO FAQ for more information.

stu1130 added 3 commits September 18, 2018 11:19

Merge branch 'master' of https://github.com/apache/incubator-mxnet

a9d40c4

Merge branch 'master' of https://github.com/apache/incubator-mxnet

429684a

update the example data format and link to each others

603b553

stu1130 requested a review from szha as a code owner September 19, 2018 23:01

fix wording

faf4994

delete the typo

540917d

marcoabreu added the pr-awaiting-review PR is waiting for code review label Sep 19, 2018

lupesko suggested changes Sep 20, 2018

View reviewed changes

sandeep-krishnamurthy approved these changes Sep 20, 2018

View reviewed changes

sandeep-krishnamurthy merged commit d4adbd2 into apache:master Sep 20, 2018

aaronmarkham reviewed Sep 20, 2018

View reviewed changes

stu1130 mentioned this pull request Sep 20, 2018

Add docstring in im2rec.py #12621

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refine the documentation of im2rec #12606

Refine the documentation of im2rec #12606

stu1130 commented Sep 19, 2018

stu1130 commented Sep 19, 2018

lupesko left a comment

lupesko Sep 20, 2018

stu1130 Sep 20, 2018 •

edited

Loading

sandeep-krishnamurthy left a comment

aaronmarkham left a comment

aaronmarkham Sep 20, 2018

aaronmarkham Sep 20, 2018

aaronmarkham Sep 20, 2018

aaronmarkham Sep 20, 2018

aaronmarkham Sep 20, 2018

		@@ -315,6 +315,8 @@ print(mx.recordio.unpack_img(s))
		You can also convert raw images into RecordIO format using the ``im2rec.py`` utility script that is provided in the MXNet [src/tools](https://github.com/dmlc/mxnet/tree/master/tools) folder.

Refine the documentation of im2rec #12606

Refine the documentation of im2rec #12606

Conversation

stu1130 commented Sep 19, 2018

Description

Checklist

Essentials

Changes

Comments

stu1130 commented Sep 19, 2018

lupesko left a comment

Choose a reason for hiding this comment

lupesko Sep 20, 2018

Choose a reason for hiding this comment

stu1130 Sep 20, 2018 • edited Loading

Choose a reason for hiding this comment

sandeep-krishnamurthy left a comment

Choose a reason for hiding this comment

aaronmarkham left a comment

Choose a reason for hiding this comment

aaronmarkham Sep 20, 2018

Choose a reason for hiding this comment

aaronmarkham Sep 20, 2018

Choose a reason for hiding this comment

aaronmarkham Sep 20, 2018

Choose a reason for hiding this comment

aaronmarkham Sep 20, 2018

Choose a reason for hiding this comment

aaronmarkham Sep 20, 2018

Choose a reason for hiding this comment

stu1130 Sep 20, 2018 •

edited

Loading