exercise-model-cards.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.7.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"papermill":{"default_parameters":{},"duration":22.765098,"end_time":"2021-02-27T00:54:36.925324","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2021-02-27T00:54:14.160226","version":"2.2.2"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**This notebook is an exercise in the [AI Ethics](https://www.kaggle.com/learn/ai-ethics) course.  You can reference the tutorial at [this link](https://www.kaggle.com/var0101/model-cards).**\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"In the tutorial, you learned how to use model cards.  In this exercise, you'll sharpen your understanding of model cards by engaging with them in a couple of scenarios.\n\n# Introduction\n\nRun the next code cell to get started.","metadata":{"papermill":{"duration":0.010515,"end_time":"2021-02-27T00:54:36.124158","exception":false,"start_time":"2021-02-27T00:54:36.113643","status":"completed"},"tags":[]}},{"cell_type":"code","source":"from learntools.core import binder\nbinder.bind(globals())\nfrom learntools.ethics.ex5 import *","metadata":{"papermill":{"duration":0.022761,"end_time":"2021-02-27T00:54:36.157779","exception":false,"start_time":"2021-02-27T00:54:36.135018","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-09-11T10:40:45.621727Z","iopub.execute_input":"2022-09-11T10:40:45.622797Z","iopub.status.idle":"2022-09-11T10:40:45.692664Z","shell.execute_reply.started":"2022-09-11T10:40:45.622675Z","shell.execute_reply":"2022-09-11T10:40:45.691513Z"},"trusted":true},"execution_count":1,"outputs":[]},{"cell_type":"markdown","source":"# Scenario A\n\nYou are the creator of the _Simple Zoom_ video editing tool, which uses AI to automatically zoom the video camera in on a presenter as they walk across a room during a presentation. You are launching the _Simple Zoom_ tool and releasing a model card along with the tool, in the interest of transparency.\n\n# 1) Audience\n\nWhich audiences should you write the model card for? Once you have thought about your answer, check it against the solution below. ","metadata":{"papermill":{"duration":0.010424,"end_time":"2021-02-27T00:54:36.178986","exception":false,"start_time":"2021-02-27T00:54:36.168562","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_1.check()","metadata":{"papermill":{"duration":0.022406,"end_time":"2021-02-27T00:54:36.212336","exception":false,"start_time":"2021-02-27T00:54:36.18993","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-09-11T10:41:25.546578Z","iopub.execute_input":"2022-09-11T10:41:25.547018Z","iopub.status.idle":"2022-09-11T10:41:25.558305Z","shell.execute_reply.started":"2022-09-11T10:41:25.546980Z","shell.execute_reply":"2022-09-11T10:41:25.557311Z"},"trusted":true},"execution_count":2,"outputs":[]},{"cell_type":"markdown","source":"# Scenario B\n\nYou are the product manager for _Presenter Pro_, a popular video and audio recording product for people delivering talks and presentations. As a new feature based on customer demand, your team has been planning to add the AI-powered ability for a single video camera to automatically track a presenter, focusing on them as they walk across the room or stage, zooming in and out automatically and continuously adjusting lighting and focus within the video frame. \n\nYou are hoping to incorporate a different company’s AI tool (called _Simple Zoom_) into your product (_Presenter Pro_). To determine whether _Simple Zoom_ is a good fit for _Presenter Pro_, you are reviewing *Simple Zoom*’s model card.\n\n# 2) Intended Use\n\nThe **Intended Use** section of the model card includes the following bullets:\n* _Simple Zoom_ is intended to be used for automatic zoom, focus, and lighting adjustment in the real-time video recording of individual presenters by a single camera\n* _Simple Zoom_ is not suitable for presentations in which there is more than one presenter or for presentations in which the presenter is partially or fully hidden at any time\n\nAs a member of the team evaluating _Simple Zoom_ for potential integration into _Presenter Pro_, you are aware that _Presenter Pro_ only supports one presenter per video.\n\nHowever, you are also aware that in some _Presenter Pro_ customers use large props in their presentation videos. Given the information in the Intended Use section of the model card, what problem do you foresee for these customers if you integrate _Simple Zoom_ into _Presenter Pro_? What are some ways in which you could address this issue?","metadata":{"papermill":{"duration":0.011583,"end_time":"2021-02-27T00:54:36.235319","exception":false,"start_time":"2021-02-27T00:54:36.223736","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Run this code cell to receive a hint\nq_2.hint()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_2.check()","metadata":{"papermill":{"duration":0.020576,"end_time":"2021-02-27T00:54:36.266949","exception":false,"start_time":"2021-02-27T00:54:36.246373","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-09-11T10:42:18.273637Z","iopub.execute_input":"2022-09-11T10:42:18.274070Z","iopub.status.idle":"2022-09-11T10:42:18.283578Z","shell.execute_reply.started":"2022-09-11T10:42:18.274030Z","shell.execute_reply":"2022-09-11T10:42:18.282514Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# 3) Factors, Evaluation Data, Metrics, and Quantitative Analyses\n\nWe'll continue with **Scenario B**, where you are the product manager for _Presenter Pro_.  Four more sections of the model card for _Simple Zoom_ are described below.\n\n**Factors**: The model card lists the following factors as potentially relevant to model performance: \n* Group Factors\n  * Self-reported gender\n  * Skin tone\n  * Self-reported age\n* Other Factors\n  * Camera angle\n  * Presenter distance from camera\n  * Camera type\n  * Lighting\n\n**Evaluation Data**: To generate the performance metrics reported in the **Quantitative Analysis** section (discussed below), the _Simple Zoom_ team used an evaluation data set of 500 presentation videos, each between two and five minutes long. The videos included both regular and unusual presentation and recording scenarios and included presenters from various demographic backgrounds.\n\n**Metrics**: Since _Simple Zoom_ model performance is subjective (involving questions like whether a zoom is of appropriate speed or smoothness; or whether a lighting adjustment is well-executed), the _Simple Zoom_ team tested the tool’s performance by asking a diverse viewer group to view _Simple Zoom_’s output videos (using the evaluation dataset’s 500 videos as inputs). Each viewer was asked to rate the quality of video editing for each video on a scale of 1 to 10, and each video’s average rating was used as a proxy for _Simple Zoom_’s performance on that video.\n\n**Quantitative Analyses**: The quantitative analyses section of the model card includes a brief summary of performance results. According to the summary, the model generally performs equally well across all the listed demographic groups (gender, skin tone and age).\n\nThe quantitative analyses section also includes interactive graphs, which allow you to view the performance of the _Simple Zoom_ tool by each factor and by intersections of ‘Group’ and ‘Other’ factors.\n\nAs a member of the team evaluating _Simple Zoom_ for potential integration into _Presenter Pro_, what are some questions you might be interested in answering and exploring via the interactive graphs?","metadata":{"papermill":{"duration":0.011538,"end_time":"2021-02-27T00:54:36.290935","exception":false,"start_time":"2021-02-27T00:54:36.279397","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Check your answer (Run this code cell to receive credit!)\nq_3.check()","metadata":{"papermill":{"duration":0.047767,"end_time":"2021-02-27T00:54:36.350911","exception":false,"start_time":"2021-02-27T00:54:36.303144","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-09-11T10:42:48.382511Z","iopub.execute_input":"2022-09-11T10:42:48.382927Z","iopub.status.idle":"2022-09-11T10:42:48.390432Z","shell.execute_reply.started":"2022-09-11T10:42:48.382889Z","shell.execute_reply":"2022-09-11T10:42:48.389721Z"},"trusted":true},"execution_count":4,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\nCongratulations!  You have reached the end of the AI Ethics course.  Once you have viewed all of the tutorials and successfully answered the questions in the exercises, you will receive a completion certificate to celebrate your hard work.  \n\nIf you enjoyed the AI Ethics course, we recommend the [**Machine Learning Explainability**](https://www.kaggle.com/learn/machine-learning-explainability) course as a next step.  To prepare for the content in this course, you should have some background in python (which you can get from the [**Python**](https://www.kaggle.com/learn/python) course) and machine learning (which you can get from the [**Intro to Machine Learning**](https://www.kaggle.com/learn/intro-to-machine-learning) course).","metadata":{"papermill":{"duration":0.01278,"end_time":"2021-02-27T00:54:36.378269","exception":false,"start_time":"2021-02-27T00:54:36.365489","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"---\n\n\n\n\n*Have questions or comments? Visit the [Learn Discussion forum](https://www.kaggle.com/learn-forum/230765) to chat with other Learners.*","metadata":{}}]}