README & Markdown files #50

athityakumar · 2017-08-09T21:59:45Z

No description provided.

athityakumar · 2017-08-09T22:01:29Z

daru-io.gemspec

+|  SciRuby Team                                 |
+*************************************************
+
+EOF


And this is how daru-io looks on installation. Added a straw to the booze to literally show "Export" from "daru". 😉

zverok · 2017-08-10T10:14:33Z

CONTRIBUTING.md

+
+# Developing the gem
+
+1. Install the extensions dependencies : Mongo, R and Redis. For any issue related to these, kindly refer to the steps mentioned in the `.travis.yml` file.


As those dependencies are not always easy to install, maybe we can soften this requirement? Like, "if you want to run entire specs suite... But to work on some particular feature..." or something like that.

zverok · 2017-08-10T10:23:17Z

README.md

+
+*Note - Each IO module has it's own set of dependencies. Have a look at the [Importers](#importers) and [Exporters](#exporters) section for dependency-specific information.*
+
+# Importers


Typically, it is only one first-level heading in Markdown ;)
And, also: maybe provide some generic info on importers? Like "When included, each importer provide DataFrame.from_<format> method, which is blah-blah"

zverok · 2017-08-10T10:24:45Z

README.md

+Daru::IO::Importers::ActiveRecord.new(args).call
+
+#! Usage from Daru::DataFrame
+Daru::DataFrame.from_activerecord(args)


I believe that two manners of usage could be placed only in importers preface above, here you can just show the most obvious way (the second one)

zverok · 2017-08-10T10:28:05Z

README.md

+```
+
+- Have a look at [this documentation](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/ActiveRecord) to know more about this Importer.
+- Dependency : activerecord gem.


Maybe instead of repeating "Have a look...." do something short and formal, like this:

ActiveRecord

Dependencies: activerecord

Usage:

require 'daru/io/importers/active_record' Daru::DataFrame.from_activerecord(connection) # and maybe some useful options example

Docs: RubyDoc.info

WDYT?

zverok · 2017-08-10T10:31:41Z

README.md

+```
+
+- Have a look at [this documentation](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/Avro) to know more about this Importer.
+- Dependency : avro gem.


And snappy?..

zverok · 2017-08-10T10:34:11Z

README.md

+- Have a look at [this documentation](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/Redis) to know more about this Importer.
+- Dependency : Standard library JSON gem, redis gem and a running `redis-server`.
+
+### SQL Importer


I believe it is in fact "sqlite file importer" or something like that?..

zverok · 2017-08-10T10:41:13Z

README.md


-## Contributing
+Contributions are always welcome. Please have a look at the [contribution guidelines](CONTRIBUTING.md) first. :tada:


I believe, that the section "creating your own importer/exporter" (as a separate lib/gem) is necessary. As well as (at the very beginning of README) a statement that it is not only a bunch of them, but also the "framework for easy creation of new importers/exporters".

zverok · 2017-08-10T10:50:53Z

daru-io.gemspec

+|  SciRuby Team                                 |
+*************************************************
+
+EOF


Please-please-please don't do this!
As a commercial programmer, who does dozens of bundle install per day, I TOTALLY HATE this "nice stuff". It takes the screen place and instead of looking at list of some updated gems I should look at some irrelevant ASCII-art.
We'll remove post-install message from daru eventually.

This option (post_install_message) is for important stuff, like on ver. 2.0.1 it can say "If you are updating from 1.5, please do ...", not for fun!!!

Ah, never saw this from this perspective. Acknowledged. 👍

zverok · 2017-08-10T10:53:01Z

lib/daru/io/exporters/rds.rb

@@ -3,6 +3,8 @@
 module Daru
  module IO
    module Exporters
+      # RDS Exporter Class, that extends +to_rds+ method to +Daru::DataFrame+
+      # instance variables


I believe it is wise to set YARD to use Markdown markup, and use it in comments.

athityakumar · 2017-08-11T20:16:34Z

Made the proposed changes. Please have a look. 😄

@zverok @v0dro - Also, from a user's perspective, I feel that YARD docs should first show information about the @param and @return before @example. But I've not been able to find a way of reordering these sections. Any insights regarding this?

zverok · 2017-08-12T18:05:14Z

But I've not been able to find a way of reordering these sections. Any insights regarding this?

I believe the only way is to make your own YARD templates, but I am not sure that rubydoc.info would respect them.

zverok · 2017-08-12T20:12:39Z

README.md


-## Installation
+While supporting various IO modules, daru-io also provides an easier way of adding more Importers / Exporters
+by means of monkey-patching. **It's strongly recommended to have a look at [this section](#tweaking-io-modules), if you're interested in tweaking and creating custom Importers / Exporters.**


I don't think that mentioning of monkey-patching is appropriate here. In fact, we "patch" almost nothing. It is a framework for defining importers/exporters and (optionally) "plugging" them as DataFrame methods.

It is important, because "monkey-patching" is considered quite questionable practice in modern Ruby, and start the explanation with proud statement about monkey-patching is bad for gem's reputation.

zverok · 2017-08-12T20:30:02Z

README.md

+    Daru::IO::Importers::ActiveRecord.new(args).call
+
+    #! Usage from Daru::DataFrame
+    Daru::DataFrame.from_activerecord(args)


Let's use more realistic examples, with meaningful arg.names. The idea that for simplest usage it should be possible to just copy-paste the example from README and go with it.

zverok · 2017-08-12T20:31:28Z

README.md

+
+- **Docs**: [rubydoc.info](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/ActiveRecord)
+- **Gem Dependencies**: `activerecord` gem
+- **Other Dependencies**: None


What's the meaning of it? If the idea is "you should not install anything else", it is a lie: to use ActiveRecord, you probably should have PostgreSQL or MySQL installed. But I believe that this line is not necessary (in form of **Other Dependencies**: None) at all.

zverok · 2017-08-12T20:32:31Z

README.md

+Imports a **Daru::DataFrame** from a **.csv** or **.csv.gz** file.
+
+- **Docs**: [rubydoc.info](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/CSV)
+- **Gem Dependencies**: Standard library `csv`, `open-uri` and `zlib` gems


Just remove the line. It communicates no information at all, just says something about internals. User definitely already has standard library and absolutely doesn't need to know which part of it you plan to require.

zverok · 2017-08-12T20:33:22Z

README.md

+Imports a **Daru::DataFrame** from a **.json** file / response.
+
+- **Docs**: [rubydoc.info](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/JSON)
+- **Gem Dependencies**: Standard library `json` gem and `jsonpath` gem


Only list jsonpath here.

zverok · 2017-08-12T20:34:05Z

README.md

+
+- **Docs**: [rubydoc.info](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Importers/SQL)
+- **Gem Dependencies**: `dbd-sqlite3`, `activerecord`, `dbi` and `sqlite3` gems
+- **Other Dependencies**: SQL


zverok · 2017-08-12T20:36:45Z

README.md

+    df.to_sql(args)
+    ```
+
+# Tweaking IO modules


I don't believe "Tweaking" is a good word here. Let's do a section named "Creating your own importers/exporters"

zverok · 2017-08-12T20:37:56Z

README.md

-TODO: Write usage instructions here
+    Behaviour of existing IO modules can also be changed according to your needs, similar to the above example.
+    For example, if the CSV Importer has to be tweaked with a faster processing gem, simply replace the above
+    (psuedo)code with `class Daru::IO::Importers::CSV`.


Is it a good suggestion? I believe it is better to suggest something like

class MyNamespace::CustomCSV < Daru::IO::Importers::CSV Daru::DataFrame.register_io_module :custom_csv, self

WDYT?..

v0dro · 2017-08-16T07:55:50Z

CONTRIBUTING.md

+
+Please proceed with a Pull Request only after you're assigned. It'd be sad if your Pull Request (and your hardwork) isn't accepted just because it isn't idealogically compatible.
+
+# Developing the gem


v0dro · 2017-08-16T07:57:01Z

LICENSE.md

@@ -1,6 +1,6 @@
 The MIT License (MIT)

-Copyright (c) 2017 athityakumar
+Copyright (c) 2017 [Athitya Kumar](https://github.com/athityakumar/)


You'll be trasferring the repo to SciRuby post GSOC so make it Athithya and SciRuby (with links).

Added SciRuby into the LICENSE. 😄

zverok

Left some comments. Sorry for pedantism.

zverok · 2017-08-17T11:05:25Z

README.md

+    Daru::IO::Importers::CSV.new('path/to/csv.gz/file', skiprows: 10, compression: :gzip).call
+
+    #! Usage from Daru::DataFrame
+    Daru::DataFrame.from_csv('path/to/csv.gz/file', skiprows: 10, compression: :gzip)


It looks a bit confuzing (like file csv.gz, and some sub-part of it named file), let's use path/to/file.csv.gz

zverok · 2017-08-17T11:46:26Z

README.md

+
+    #! Usage from Daru::DataFrame
+    require 'daru/io/importers/excel'
+    Daru::DataFrame.from_excel('path/to/xlsx/file', sheet: 2, skiprows: 10, skipcols: 2)


...and here and everywhere in likewise cases path/to/file.xlsx

zverok · 2017-08-17T11:47:53Z

README.md

+    Daru::IO::Importers::HTML.new('website/domain/url', match: 'market', name: 'Shares analysis').call
+
+    #! Usage from Daru::DataFrame
+    Daru::DataFrame.from_html('website/domain/url', match: 'market', name: 'Shares analysis')


I'd say

# local Daru::DataFrame.from_html('path/to/file.html', match: 'market', name: 'Shares analysis') # or remote Daru::DataFrame.from_html('https://some/url/with/tables', match: 'market', name: 'Shares analysis')

...this showcase is MUCH more useful than adding obvious Daru::IO::Importers::HTML.new().call to each importer, which I am against.

zverok · 2017-08-17T11:50:00Z

README.md

+
+[(Go to Table of Contents)](#table-of-contents)
+
+Imports a **Daru::DataFrame** from a **.dat** plaintext file.


Maybe some link/explanation about expected format of the file?

zverok · 2017-08-17T11:51:11Z

README.md

+    Daru::IO::Importers::SQL.new('path/to/sqlite.db/file', 'SELECT * FROM test').call
+
+    #! Usage from Daru::DataFrame
+    Daru::DataFrame.from_sql('path/to/sqlite.db/file', 'SELECT * FROM test')


... or DBI connection ;)

zverok · 2017-08-17T11:53:13Z

README.md

+
+[(Go to Table of Contents)](#table-of-contents)
+
+Exports a **Daru::DataFrame** into an SQL table.


I'd say "To database table through DBI connection". And, don't we export to SQLite files?..

zverok · 2017-08-17T11:53:36Z

README.md

+
+- **Docs**: [rubydoc.info](http://www.rubydoc.info/github/athityakumar/daru-io/master/Daru/IO/Exporters/SQL)
+- **Gem Dependencies**: `dbd-sqlite3`, `dbi` and `sqlite3` gems
+- **Other Dependencies**: SQL


What is "SQL dependency"?.. "You should have SQL installed"?

zverok · 2017-08-17T11:54:37Z

daru-io.gemspec

+  Daru-IO is a plugin-gem to Daru gem, which stands for Data Analysis in RUby. Daru-IO extends support for many Import and Export methods of Daru::DataFrame. This gem is intended to help Rubyists who are into Data Analysis or Web Development, by serving as a general purpose conversion library that takes input in one format (say, JSON) and converts it another format (say, Avro) while also making it incredibly easy to getting started on analyzing data with daru.
+
+  While supporting various IO modules, daru-io also provides an easier way of adding new Importers / Exporters.
+MSG


I believe it is over-complication. Why?

zverok · 2017-08-17T11:54:58Z

lib/daru/io/base.rb

@@ -3,15 +3,41 @@

 module Daru
  module IO
+    # Base IO Class that contains generic helper methods, to be
+    # used by other **Importers::Base** and **Exporters::Base** via inheritence


{Importers::Base} would be auto-linked.

zverok · 2017-08-17T11:55:44Z

lib/daru/io/exporters/avro.rb

      class Avro < Base
        Daru::DataFrame.register_io_module :to_avro, self

-        # Exports +Daru::DataFrame+ to an Avro file.
+        # Exports **Daru::DataFrame** to an Avro file.


`Daru::DataFrame`

would be prettier, probably. And above too.

The to_format method is pending, and is quite ambigous as in , returning which format instance would make more sense. Also, the YARD docs is yet to be updated as it's been updated for markdown support in PR #50 and would be better to proceed after that PR is merged.

zverok

Let's merge it! :allthethings:

athityakumar added 3 commits August 10, 2017 02:52

Starts adding better markdown files

f4c7651

Minor changes in README

1108a66

Adds post-install message in gemspec file

4bc14b1

athityakumar commented Aug 9, 2017

View reviewed changes

athityakumar added 5 commits August 10, 2017 04:05

Documents #register_io_module with linkages

10822a4

Documents #optional_gem method

13f3cd9

Documents Exporters::Base#initialize method

a328841

Documents Importers::Base#guess_parse method

4bf7dbc

Documents IO classes

2ed67b7

athityakumar force-pushed the documentation branch 2 times, most recently from e300e0e to aa983c8 Compare August 10, 2017 00:52

Reorders rubocop rules in alphabetic order

aa983c8

zverok requested changes Aug 10, 2017

View reviewed changes

athityakumar added 2 commits August 11, 2017 07:56

Adds changes as per review

fbe0d3a

Converts yard comments to github flavoured markdown

2797326

zverok requested changes Aug 12, 2017

View reviewed changes

athityakumar added 3 commits August 13, 2017 15:30

More review suggestions

00385d3

Merge branch 'master' into documentation

d25d18e

Adds small note to HTML Importer

a173439

v0dro reviewed Aug 16, 2017

View reviewed changes

Adds SciRuby org into LICENSE

2a89199

zverok reviewed Aug 17, 2017

View reviewed changes

athityakumar added 3 commits August 19, 2017 15:50

Adds a couple more changes into YARD Doc & README

fb47ece

Links RData IO modules documentation to contribution guidelines

5448524

Adds minor correction to plaintext importer

8895628

athityakumar mentioned this pull request Aug 26, 2017

Adds from, read, write, to_s methods #52

Merged

zverok approved these changes Aug 26, 2017

View reviewed changes

athityakumar merged commit e8de525 into master Aug 26, 2017

athityakumar mentioned this pull request Aug 26, 2017

Markdown files #14

Closed

4 tasks


		# Developing the gem

		1. Install the extensions dependencies : Mongo, R and Redis. For any issue related to these, kindly refer to the steps mentioned in the `.travis.yml` file.


		Note - Each IO module has it's own set of dependencies. Have a look at the [Importers](#importers) and [Exporters](#exporters) section for dependency-specific information.

		# Importers


		## Contributing
		Contributions are always welcome. Please have a look at the [contribution guidelines](CONTRIBUTING.md) first. :tada:


		Please proceed with a Pull Request only after you're assigned. It'd be sad if your Pull Request (and your hardwork) isn't accepted just because it isn't idealogically compatible.

		# Developing the gem


		[(Go to Table of Contents)](#table-of-contents)

		Imports a Daru::DataFrame from a .dat plaintext file.


		[(Go to Table of Contents)](#table-of-contents)

		Exports a Daru::DataFrame into an SQL table.

README & Markdown files #50

README & Markdown files #50

Conversation

athityakumar commented Aug 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ActiveRecord

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

athityakumar commented Aug 11, 2017

zverok commented Aug 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zverok left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zverok left a comment

Choose a reason for hiding this comment