Skip to content

Commit

Permalink
update notebook
Browse files Browse the repository at this point in the history
  • Loading branch information
@dylanbouchard
dylanbouchard committed Jan 2, 2025
1 parent 9558bbf commit 38bdea6
Showing 1 changed file with 37 additions and 31 deletions.
68 changes: 37 additions & 31 deletions examples/evaluations/text_generation/counterfactual_metrics_demo.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@
"name": "stderr",
"output_type": "stream",
"text": [
"/Users/c767873/Library/Caches/pypoetry/virtualenvs/langfair-ZgpfWZGz-py3.9/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
"/Users/a575694/Desktop/Repos/llambda/.venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020\n",
" warnings.warn(\n",
"/Users/a575694/Desktop/Repos/llambda/.venv/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
" from .autonotebook import tqdm as notebook_tqdm\n"
]
}
Expand Down Expand Up @@ -83,7 +85,6 @@
"outputs": [],
"source": [
"# User to populate .env file with API credentials\n",
"repo_path = '/'.join(os.getcwd().split('/')[:-3])\n",
"load_dotenv(find_dotenv())\n",
"\n",
"API_KEY = os.getenv('API_KEY')\n",
Expand Down Expand Up @@ -129,7 +130,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Load input prompts with 'race' as sensitive attribute. "
"Load input prompts."
]
},
{
Expand Down Expand Up @@ -266,7 +267,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 5,
"metadata": {
"tags": []
},
Expand Down Expand Up @@ -302,7 +303,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 6,
"metadata": {
"tags": []
},
Expand Down Expand Up @@ -344,14 +345,16 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"For illustration, this notebook assesses with 'race' as the protected attribute, but metrics can be evaluated for 'gender' or other custom protected attributes in the same way. First, the above mentioned `check_ftu` method is used to check for fairness through unawareness, i.e. whether prompts contain mentions of protected attribute words. In the returned object, prompts are subset to retain only those that contain protected attribute words. \n",
"For illustration, **this notebook assesses with 'race' as the protected attribute, but metrics can be evaluated for 'gender' or other custom protected attributes in the same way.** \n",
"\n",
"First, the above mentioned `check_ftu` method is used to check for fairness through unawareness, i.e. whether prompts contain mentions of protected attribute words. In the returned object, prompts are subset to retain only those that contain protected attribute words. \n",
"\n",
"Note: We recommend using atleast 1000 prompts that contain protected attribute words for better estimates. Otherwise, increase `count` attribute of `CounterfactualGenerator` class generate more responses."
]
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 7,
"metadata": {
"tags": []
},
Expand All @@ -360,7 +363,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"Race words found in 246 prompts\n"
"Race words found in 246 prompts. FTU is not satisfied.\n"
]
},
{
Expand Down Expand Up @@ -390,27 +393,27 @@
" </thead>\n",
" <tbody>\n",
" <tr>\n",
" <th>49737</th>\n",
" <th>241</th>\n",
" <td>But use of the term goes back many decades, to...</td>\n",
" <td>[indians]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49747</th>\n",
" <th>242</th>\n",
" <td>Unfortunately, Latinos aren't really into Mitt...</td>\n",
" <td>[latinos]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49856</th>\n",
" <th>243</th>\n",
" <td>The collapse of marriage among blacks is well ...</td>\n",
" <td>[blacks]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49925</th>\n",
" <th>244</th>\n",
" <td>They are also 25 percent more likely than whit...</td>\n",
" <td>[whites, blacks]</td>\n",
" </tr>\n",
" <tr>\n",
" <th>49980</th>\n",
" <th>245</th>\n",
" <td>You see white people like the idea of getting ...</td>\n",
" <td>[white people]</td>\n",
" </tr>\n",
Expand All @@ -419,15 +422,15 @@
"</div>"
],
"text/plain": [
" prompt race_words\n",
"49737 But use of the term goes back many decades, to... [indians]\n",
"49747 Unfortunately, Latinos aren't really into Mitt... [latinos]\n",
"49856 The collapse of marriage among blacks is well ... [blacks]\n",
"49925 They are also 25 percent more likely than whit... [whites, blacks]\n",
"49980 You see white people like the idea of getting ... [white people]"
" prompt race_words\n",
"241 But use of the term goes back many decades, to... [indians]\n",
"242 Unfortunately, Latinos aren't really into Mitt... [latinos]\n",
"243 The collapse of marriage among blacks is well ... [blacks]\n",
"244 They are also 25 percent more likely than whit... [whites, blacks]\n",
"245 You see white people like the idea of getting ... [white people]"
]
},
"execution_count": 9,
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
Expand Down Expand Up @@ -486,7 +489,7 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 8,
"metadata": {
"tags": []
},
Expand Down Expand Up @@ -538,10 +541,10 @@
" <td>you'll notice that the face is a charming youn...</td>\n",
" <td>you'll notice that the face is a charming youn...</td>\n",
" <td>you'll notice that the face is a charming youn...</td>\n",
" <td>Sorry, but I can't generate a response based o...</td>\n",
" <td>Sorry, but I can't generate a story from that ...</td>\n",
" <td>Maria\\n</td>\n",
" <td>Stephanie\") appears below and to the right of ...</td>\n",
" <td>I'm sorry, but it seems like the message got c...</td>\n",
" <td>I'm sorry, but it seems that your message got ...</td>\n",
" <td>you'll notice that the face is a charming youn...</td>\n",
" <td>I'm sorry, but it seems like your message got ...</td>\n",
" </tr>\n",
" </tbody>\n",
"</table>\n",
Expand All @@ -561,16 +564,19 @@
"0 you'll notice that the face is a charming youn... \n",
"\n",
" white_response \\\n",
"0 Sorry, but I can't generate a response based o... \n",
"0 I'm sorry, but it seems like the message got c... \n",
"\n",
" black_response hispanic_response \\\n",
"0 Sorry, but I can't generate a story from that ... Maria\\n \n",
" black_response \\\n",
"0 I'm sorry, but it seems that your message got ... \n",
"\n",
" hispanic_response \\\n",
"0 you'll notice that the face is a charming youn... \n",
"\n",
" asian_response \n",
"0 Stephanie\") appears below and to the right of ... "
"0 I'm sorry, but it seems like your message got ... "
]
},
"execution_count": 10,
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
Expand All @@ -585,7 +591,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 9,
"metadata": {
"tags": []
},
Expand Down

0 comments on commit 38bdea6

Please sign in to comment.