datascience.md

The datascience Package

The datascience package is an open source Python package that helps make programming more accessible to all students, regardless of background. As a pedagogical aid, the package is designed to help students more intuitively conduct data science techniques without first spending considerable time directly learning more complex tools such as pandas or matplotlib. At Berkeley, these other packages are introduced in further upper-division coursework such as Data 100.

Technical details

The datascience package was built with the main goal to teach students about working with tables and visualizations in an introductory data science setting. It was inspired by techniques in SQL, pandas, and R data frames, and follows a more natural langauge programming design to have a more intuitive way in syntax.

The package is built on built-in Python data structures, with several dependencies:

• NumPy: a tool for numerical computing and linear algebra. The datascience package relies on numpy arrays as its primary data structure; for example, each column in datascience Table objects are numpy arrays. Often, many numpy functions are also separtely introduced in the course, such as np.mean or np.append.
• SciPy: a set of tools for scientific computing. The minimize function, used to minimize RMSE, uses the optimize module from scipy.
• Matplotlib: a tool for visualization. Plotting is directly done in datascience by calling plotting functions on Table objects. Notably, tweaking plots such as renaming titles or adjusting axis shape are abstracted away from students.
• pandas: the more industry standard tool for data manipulation and analysis. Although pandas is not a significant dependency, datascience supports conversion between its Table objects and pandas dataframes.

The datascience python package was written by Berkeley professors John DeNero and David Culler, as well as students Sam Lau and Alvin Wan. The full documentation to the datascience package can be found here, but students typically only need the Python Reference Guide for all the functions that are used widely in Data 8.

Pedagogical Choices

One large barrier to entry in doing data science for many students is the coding knowledge required. Since Data 8 was designed to be highly accessible to students of all backgrounds, the datascience package was thus created to help make the programming part of the course more accessible to students with no coding background by removing syntax complexities. However, this decision comes with a profound trade-off: the package loses computational flexibility and power for increased ease of understanding and usage compared to industry-standard tools such as pandas. This trade-off was acceptable for teaching Data 8, as datasets and their associated computation are typically not too large (<100 MB), and the computational flexibility required is limited to within the scope of the course.

Overall, Data 8 emphasizes developing computational thinking skills over details in the specific syntax. This training allows students to more seamlessly transition to other more complex packages after Data 8.

One limitation from using the datascience package is that it does not support a wide range of data cleaning procedures. Data 8 abstracts away methods in data cleaning, which will instead be taught in Data 100. As such, students typically receive well-formed data without missing values in Data 8. However, if you plan on placing a larger focus data cleaning or more advanced data manipulation procedures in your course, using pandas may perhaps be more appropriate.