[REVIEW]: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases #7570

editorialbot · 2024-12-04T20:56:49Z

Submitting author: @dylanbouchard (Dylan Bouchard)
Repository: https://github.com/cvs-health/langfair
Branch with paper.md (empty if default branch): joss_paper
Version: v0.3.1
Editor: @crvernon
Reviewers: @xavieryao, @emily-sexton
Archive: 10.5281/zenodo.14622998

Status

Status badge code:

HTML: <a href="https://joss.theoj.org/papers/0fbc09de7fa4e873ac68c3cb30afdd66"><img src="https://joss.theoj.org/papers/0fbc09de7fa4e873ac68c3cb30afdd66/status.svg"></a>
Markdown: [![status](https://joss.theoj.org/papers/0fbc09de7fa4e873ac68c3cb30afdd66/status.svg)](https://joss.theoj.org/papers/0fbc09de7fa4e873ac68c3cb30afdd66)

Reviewers and authors:

Please avoid lengthy details of difficulties in the review thread. Instead, please create a new issue in the target repository and link to those issues (especially acceptance-blockers) by leaving comments in the review thread below. (For completists: if the target issue tracker is also on GitHub, linking the review thread in the issue or vice versa will create corresponding breadcrumb trails in the link target.)

Reviewer instructions & questions

@xavieryao & @emily-sexton, your review will be checklist based. Each of you will have a separate checklist that you should update when carrying out your review.
First of all you need to run this command in a separate comment to create the checklist:

@editorialbot generate my checklist

The reviewer guidelines are available here: https://joss.readthedocs.io/en/latest/reviewer_guidelines.html. Any questions/concerns please let @crvernon know.

✨ Please start on your review when you are able, and be sure to complete your review in the next six weeks, at the very latest ✨

Checklists

📝 Checklist for @emily-sexton

📝 Checklist for @xavieryao

The text was updated successfully, but these errors were encountered:

editorialbot · 2024-12-04T20:56:53Z

Hello humans, I'm @editorialbot, a robot that can help you with some common editorial tasks.

For a list of things I can do to help you, just type:

@editorialbot commands

For example, to regenerate the paper pdf after making changes in the paper's md or bib files, type:

@editorialbot generate pdf

editorialbot · 2024-12-04T20:57:33Z

Software report:

github.com/AlDanial/cloc v 1.90  T=0.41 s (230.7 files/s, 276386.2 lines/s)
-------------------------------------------------------------------------------
Language                     files          blank        comment           code
-------------------------------------------------------------------------------
JSON                            11              0              0          99452
Python                          58            695           1920           3399
Jupyter Notebook                11              0           6138            834
TeX                              2             93             78            474
Markdown                         7            156              0            415
TOML                             1              5              0             57
YAML                             3              9             10             52
CSV                              2              0              0              2
-------------------------------------------------------------------------------
SUM:                            95            958           8146         104685
-------------------------------------------------------------------------------

Commit count by author:

    72	Dylan Bouchard
     6	Viren Bajaj
     5	zeya30
     4	Mohit Singh Chauhan
     2	David Skarbrevik
     2	Zeya Ahmad
     2	virenbajaj
     1	Vasistha

editorialbot · 2024-12-04T20:57:37Z

Paper file info:

📄 Wordcount for paper.md is 1582

✅ The paper includes a Statement of need section

editorialbot · 2024-12-04T20:57:38Z

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.1162/tacl_a_00240 is OK
- 10.1145/3442188.3445924 is OK
- 10.18653/v1/2021.naacl-main.191 is OK
- 10.18653/v1/2022.findings-acl.165 is OK
- 10.18653/v1/2020.findings-emnlp.311 is OK
- 10.1145/3576840.3578295 is OK
- 10.1016/j.simpa.2024.100619 is OK
- 10.5281/zenodo.12608602 is OK
- 10.18653/v1/N19-3002 is OK
- 10.1145/3604915.3608860 is OK
- 10.18653/v1/2022.naacl-main.122 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: Gender Bias in Coreference Resolution
- No DOI given, and none found for title: Unmasking Contextual Stereotypes: Measuring and Mi...
- No DOI given, and none found for title: TrustGPT: A Benchmark for Trustworthy and Responsi...
- No DOI given, and none found for title: Holistic Evaluation of Language Models
- No DOI given, and none found for title: DecodingTrust: A Comprehensive Assessment of Trust...
- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...
- No DOI given, and none found for title: Beyond the Imitation Game: Quantifying and extrapo...
- No DOI given, and none found for title: TrustLLM: Trustworthiness in Large Language Models
- No DOI given, and none found for title: AI Fairness 360:  An Extensible Toolkit for Detect...
- No DOI given, and none found for title: Fairlearn: Assessing and Improving Fairness of AI ...
- No DOI given, and none found for title: Aequitas: A Bias and Fairness Audit Toolkit
- No DOI given, and none found for title: GitHub - tensorflow/fairness-indicators: Tensorflo...
- No DOI given, and none found for title: LiFT: A Scalable Framework for Measuring Fairness ...
- No DOI given, and none found for title: An Actionable Framework for Assessing Bias and Fai...
- No DOI given, and none found for title: Towards Auditing Large Language Models: Improving ...
- No DOI given, and none found for title: Counterfactual Fairness
- No DOI given, and none found for title: Bias and Fairness in Large Language Models: A Surv...
- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...

❌ MISSING DOIs

- 10.18653/v1/n18-2003 may be a valid DOI for title: Gender Bias in Coreference Resolution: Evaluation ...
- 10.18653/v1/2021.findings-emnlp.211 may be a valid DOI for title: Collecting a Large-Scale Gender Bias Dataset for C...
- 10.18653/v1/2021.acl-long.416 may be a valid DOI for title: StereoSet: Measuring stereotypical bias in pretrai...
- 10.18653/v1/2020.emnlp-main.154 may be a valid DOI for title: CrowS-Pairs: A Challenge Dataset for Measuring Soc...
- 10.18653/v1/2023.acl-long.507 may be a valid DOI for title: WinoQueer: A Community-in-the-Loop Benchmark for A...
- 10.18653/v1/2021.acl-long.151 may be a valid DOI for title: RedditBias: A Real-World Resource for Bias Evaluat...
- 10.18653/v1/2022.emnlp-main.646 may be a valid DOI for title: Perturbation Augmentation for Fairer NLP
- 10.18653/v1/s18-2005 may be a valid DOI for title: Examining Gender and Race Bias in Two Hundred Sent...
- 10.18653/v1/2020.findings-emnlp.301 may be a valid DOI for title: RealToxicityPrompts: Evaluating Neural Toxic Degen...
- 10.1109/tvcg.2019.2934619 may be a valid DOI for title: The What-If Tool: Interactive Probing of Machine L...
- 10.18653/v1/2020.findings-emnlp.7 may be a valid DOI for title: Reducing Sentiment Bias in Language Models via Cou...
- 10.1145/2783258.2783311 may be a valid DOI for title: Certifying and removing disparate impact
- 10.18653/v1/2021.acl-long.150 may be a valid DOI for title: Intrinsic Bias Metrics Do Not Correlate with Appli...

❌ INVALID DOIs

- None

editorialbot · 2024-12-04T20:57:38Z

License info:

🟡 License found: Other (Check here for OSI approval)

crvernon · 2024-12-04T20:58:22Z

👋 @dylanbouchard, @xavieryao, and @emily-sexton - This is the review thread for the paper. All of our communications will happen here from now on.

Please read the "Reviewer instructions & questions" in the first comment above.

Both reviewers have checklists at the top of this thread (in that first comment) with the JOSS requirements. As you go over the submission, please check any items that you feel have been satisfied. There are also links to the JOSS reviewer guidelines.

The JOSS review is different from most other journals. Our goal is to work with the authors to help them meet our criteria instead of merely passing judgment on the submission. As such, the reviewers are encouraged to submit issues and pull requests on the software repository. When doing so, please mention #7570 so that a link is created to this thread (and I can keep an eye on what is happening). Please also feel free to comment and ask questions on this thread. In my experience, it is better to post comments/questions/suggestions as you come across them instead of waiting until you've reviewed the entire package.

We aim for the review process to be completed within about 4-6 weeks but please make a start well ahead of this as JOSS reviews are by their nature iterative and any early feedback you may be able to provide to the author will be very helpful in meeting this schedule.

editorialbot · 2024-12-04T20:58:58Z

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

emily-sexton · 2024-12-06T14:14:25Z

Review checklist for @emily-sexton

Conflict of interest

I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

I confirm that I read and will adhere to the JOSS code of conduct.

General checks

Repository: Is the source code for this software available at the https://github.com/cvs-health/langfair?
License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
Contribution and authorship: Has the submitting author (@dylanbouchard) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

Installation: Does installation proceed as outlined in the documentation?
Functionality: Have the functional claims of the software been confirmed?
Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
State of the field: Do the authors describe how this software compares to other commonly-used packages?
Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

xavieryao · 2024-12-17T00:37:25Z

Review checklist for @xavieryao

Conflict of interest

I confirm that I have read the JOSS conflict of interest (COI) policy and that: I have no COIs with reviewing this work or that any perceived COIs have been waived by JOSS for the purpose of this review.

Code of Conduct

I confirm that I read and will adhere to the JOSS code of conduct.

General checks

Repository: Is the source code for this software available at the https://github.com/cvs-health/langfair?
License: Does the repository contain a plain-text LICENSE or COPYING file with the contents of an OSI approved software license?
Contribution and authorship: Has the submitting author (@dylanbouchard) made major contributions to the software? Does the full list of paper authors seem appropriate and complete?
Substantial scholarly effort: Does this submission meet the scope eligibility described in the JOSS guidelines
Data sharing: If the paper contains original data, data are accessible to the reviewers. If the paper contains no original data, please check this item.
Reproducibility: If the paper contains original results, results are entirely reproducible by reviewers. If the paper contains no original results, please check this item.
Human and animal research: If the paper contains original data research on humans subjects or animals, does it comply with JOSS's human participants research policy and/or animal research policy? If the paper contains no such data, please check this item.

Functionality

Installation: Does installation proceed as outlined in the documentation?
Functionality: Have the functional claims of the software been confirmed?
Performance: If there are any performance claims of the software, have they been confirmed? (If there are no claims, please check off this item.)

Documentation

A statement of need: Do the authors clearly state what problems the software is designed to solve and who the target audience is?
Installation instructions: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.
Example usage: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems).
Functionality documentation: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?
Automated tests: Are there automated tests or manual steps described so that the functionality of the software can be verified?
Community guidelines: Are there clear guidelines for third parties wishing to 1. Contribute to the software 2. Report issues or problems with the software 3. Seek support

Software paper

Summary: Has a clear description of the high-level functionality and purpose of the software for a diverse, non-specialist audience been provided?
A statement of need: Does the paper have a section titled 'Statement of need' that clearly states what problems the software is designed to solve, who the target audience is, and its relation to other work?
State of the field: Do the authors describe how this software compares to other commonly-used packages?
Quality of writing: Is the paper well written (i.e., it does not require editing for structure, language, or writing quality)?
References: Is the list of references complete, and is everything cited appropriately that should be cited (e.g., papers, datasets, software)? Do references in the text use the proper citation syntax?

emily-sexton · 2024-12-21T16:53:40Z

@dylanbouchard - I can see that there's a reference, 'Beyond the imitation game...' that has the author listed as 'authors, B.' When I go to the reference, it looks like there are hundreds of authors. Is this a way of referencing a paper with too many authors to list?
Also, were the missing DOIs mentioned about addressed?
@crvernon

dylanbouchard · 2024-12-21T20:51:25Z

@dylanbouchard - I can see that there's a reference, 'Beyond the imitation game...' that has the author listed as 'authors, B.' When I go to the reference, it looks like there are hundreds of authors. Is this a way of referencing a paper with too many authors to list? Also, were the missing DOIs mentioned about addressed? @crvernon

@emily-sexton, that is my understanding yes. All of the references are auto-populated using bibtex, so that would be my guess.

emily-sexton · 2024-12-22T17:28:44Z

@dylanbouchard - I'm having troubles installing langfair because I have Python 3.13.1 and looks like langfair can only be used with Python <3.12, >=3.9. I'm trying to install an old version of python in a virtual environment but the best documentation i can find on stack overflow is many years old (https://stackoverflow.com/questions/5506110/is-it-possible-to-install-another-version-of-python-to-virtualenv) and I'm struggling.

Can you give some guidance on how best to install langfair if I have a newer version of python? Should I be installing an older version of python in a virtual environment? If so, do you know the best way to do that? Thanks!!

dylanbouchard · 2024-12-22T18:06:06Z

@emily-sexton I have written a new answer to your question below. This can be done with venv as follows:

python3.9 -m venv myenv

xavieryao · 2024-12-29T23:52:23Z

@dylanbouchard great work! This sets an example of high quality (open source) software project.

Some minor issues that can be easily patched:

Minor text errors in the readme and a notebook: Fix broken links in README and copy-paste errors in example notebook cvs-health/langfair#81
Omit the brackets when referring to the author as part of a sentence: “For a case study on writers block, see Upper (1974).” Such as at line 71 of the paper. See JOSS guideline.
Paper line 65: can you add an (informal) introduction of FTU? I understand that you have formal definitions in the technical companion, but it would be more helpful for the readers.
Paper line 118: explicitly mention that the steps are what have been described previously?
Add the missing DOIs if possible.

My understanding is that the metrics for recommendation and classification are not specific to LLMs. I see that it is convenient to have them in one package, as these are also applications of LLMs. Could you add a discussion of existing repositories that offer those generic fairness metrics? Examples:
https://github.com/tensorflow/fairness-indicators
https://github.com/Trusted-AI/AIF360
https://github.com/amazon-science/generalized-fairness-metrics?tab=readme-ov-file

Is there a clearer pointer to "off-the-shelf FTU check" (line 67) other than in an example notebook? cvs-health/langfair#82

crvernon · 2024-12-30T15:25:13Z

@editorialbot generate pdf

editorialbot · 2024-12-30T15:26:43Z

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

crvernon · 2024-12-30T15:30:08Z

@editorialbot remind me in 1 week

@dylanbouchard - I can see that there's a reference, 'Beyond the imitation game...' that has the author listed as 'authors, B.' When I go to the reference, it looks like there are hundreds of authors. Is this a way of referencing a paper with too many authors to list? Also, were the missing DOIs mentioned about addressed? @crvernon

@emily-sexton, that is my understanding yes. All of the references are auto-populated using bibtex, so that would be my guess.

Let me check on this. I'll provide feedback when I make sure that we have a consistent protocol on this type of publication. Thanks!

editorialbot · 2024-12-30T15:30:10Z

Reminder set for @crvernon in 1 week

dylanbouchard · 2024-12-31T19:27:24Z

@dylanbouchard great work! This sets an example of high quality (open source) software project.

Some minor issues that can be easily patched:

Minor text errors in the readme and a notebook: Fix broken links in README and copy-paste errors in example notebook cvs-health/langfair#81

Omit the brackets when referring to the author as part of a sentence: “For a case study on writers block, see Upper (1974).” Such as at line 71 of the paper. See JOSS guideline.

Paper line 65: can you add an (informal) introduction of FTU? I understand that you have formal definitions in the technical companion, but it would be more helpful for the readers.

Paper line 118: explicitly mention that the steps are what have been described previously?

Add the missing DOIs if possible.

My understanding is that the metrics for recommendation and classification are not specific to LLMs. I see that it is convenient to have them in one package, as these are also applications of LLMs. Could you add a discussion of existing repositories that offer those generic fairness metrics? Examples: https://github.com/tensorflow/fairness-indicators https://github.com/Trusted-AI/AIF360 https://github.com/amazon-science/generalized-fairness-metrics?tab=readme-ov-file

Is there a clearer pointer to "off-the-shelf FTU check" (line 67) other than in an example notebook? cvs-health/langfair#82

Thank you @xavieryao! Really appreciate the helpful suggestions. Working on these changes now!

xavieryao · 2025-01-01T03:33:10Z

@dylanbouchard Thank you for the quick responses. All of my concerns have been adequately addressed and I think both the paper and the project are of great quality.

@crvernon I have no further comments and would recommend Accept.

Happy new year.

dylanbouchard · 2025-01-01T15:02:04Z

Thank you very much @xavieryao !

dylanbouchard · 2025-01-01T23:31:52Z

@editorialbot generate pdf

editorialbot · 2025-01-01T23:33:29Z

👉📄 Download article proof 📄 View article proof on GitHub 📄 👈

dylanbouchard · 2025-01-02T19:42:09Z

@crvernon two questions:

Should the submission be updated to the version that includes the updates recommended and approved by @xavieryao, i.e. v0.3.1?
Is there a preferred way to handle text contained in backticks from spilling over the margin? This appears to be happening in line 55 with the mention of CounterfactualGenerator

crvernon · 2025-01-23T14:47:29Z

@editorialbot recommend-accept

editorialbot · 2025-01-23T14:47:32Z

Attempting dry run of processing paper acceptance...

editorialbot · 2025-01-23T14:47:57Z

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.48550/arXiv.1804.09301 is OK
- 10.48550/arXiv.1804.06876 is OK
- 10.1162/tacl_a_00240 is OK
- 10.48550/arXiv.2109.03858 is OK
- 10.48550/arXiv.2004.09456 is OK
- 10.48550/arXiv.2010.14534 is OK
- 10.48550/arXiv.2010.00133 is OK
- 10.48550/arXiv.2306.15087 is OK
- 10.48550/arXiv.2106.03521 is OK
- 10.48550/arXiv.2205.12586 is OK
- 10.18653/v1/S18-2005 is OK
- 10.18653/v1/2020.findings-emnlp.301 is OK
- 10.1145/3442188.3445924 is OK
- 10.48550/arXiv.2306.11507 is OK
- 10.18653/v1/2021.naacl-main.191 is OK
- 10.18653/v1/2022.findings-acl.165 is OK
- 10.18653/v1/2020.findings-emnlp.311 is OK
- 10.1145/3576840.3578295 is OK
- 10.48550/arXiv.2211.09110 is OK
- 10.48550/arXiv.2306.11698 is OK
- 10.1016/j.simpa.2024.100619 is OK
- 10.48550/arXiv.2206.04615 is OK
- 10.5281/zenodo.12608602 is OK
- 10.48550/arXiv.2401.05561 is OK
- 10.48550/arXiv.1810.01943 is OK
- 10.48550/arXiv.1811.05577 is OK
- 10.1109/TVCG.2019.2934619 is OK
- 10.1145/3340531.3412705 is OK
- 10.48550/arXiv.2407.10853 is OK
- 10.18653/v1/N19-3002 is OK
- 10.48550/arXiv.2311.14126 is OK
- 10.48550/arXiv.1703.06856 is OK
- 10.48550/arXiv.2309.00770 is OK
- 10.18653/v1/2020.findings-emnlp.7 is OK
- 10.1145/3604915.3608860 is OK
- 10.1145/2783258.2783311 is OK
- 10.18653/v1/2021.acl-long.150 is OK
- 10.18653/v1/2022.naacl-main.122 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...
- No DOI given, and none found for title: Fairlearn: Assessing and Improving Fairness of AI ...
- No DOI given, and none found for title: GitHub - tensorflow/fairness-indicators: Tensorflo...
- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

editorialbot · 2025-01-23T14:49:10Z

⚠️ Error preparing paper acceptance. The generated XML metadata file is invalid.

Element citation, attribute 'key': [facet 'maxLength'] The value 'Arshaan_Nazir_and_Thadaka_Kalyan_Chakravarthy_and_David_Amore_Cecchini_and_Thadaka_Kalyan_Chakravarthy_and_Rakshit_Khajuria_and_Prikshit_Sharma_and_Ali_Tarik_Mirik_and_Veysel_Kocaman_and_David_Talby_LangTest_A_comprehensive_2024' has a length of '228'; this exceeds the allowed maximum length of '128'.

dylanbouchard · 2025-01-23T14:53:34Z

⚠️ Error preparing paper acceptance. The generated XML metadata file is invalid.

Element citation, attribute 'key': [facet 'maxLength'] The value 'Arshaan_Nazir_and_Thadaka_Kalyan_Chakravarthy_and_David_Amore_Cecchini_and_Thadaka_Kalyan_Chakravarthy_and_Rakshit_Khajuria_and_Prikshit_Sharma_and_Ali_Tarik_Mirik_and_Veysel_Kocaman_and_David_Talby_LangTest_A_comprehensive_2024' has a length of '228'; this exceeds the allowed maximum length of '128'.

@crvernon I should be able to just use a different key for this one, let me see

dylanbouchard · 2025-01-23T14:57:01Z

@crvernon I have changed the key to be shorter and checked that the pdf compiles correctly. This should be all set now.

crvernon · 2025-01-23T15:06:45Z

@editorialbot recommend-accept

editorialbot · 2025-01-23T15:06:47Z

Attempting dry run of processing paper acceptance...

editorialbot · 2025-01-23T15:07:22Z

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.48550/arXiv.1804.09301 is OK
- 10.48550/arXiv.1804.06876 is OK
- 10.1162/tacl_a_00240 is OK
- 10.48550/arXiv.2109.03858 is OK
- 10.48550/arXiv.2004.09456 is OK
- 10.48550/arXiv.2010.14534 is OK
- 10.48550/arXiv.2010.00133 is OK
- 10.48550/arXiv.2306.15087 is OK
- 10.48550/arXiv.2106.03521 is OK
- 10.48550/arXiv.2205.12586 is OK
- 10.18653/v1/S18-2005 is OK
- 10.18653/v1/2020.findings-emnlp.301 is OK
- 10.1145/3442188.3445924 is OK
- 10.48550/arXiv.2306.11507 is OK
- 10.18653/v1/2021.naacl-main.191 is OK
- 10.18653/v1/2022.findings-acl.165 is OK
- 10.18653/v1/2020.findings-emnlp.311 is OK
- 10.1145/3576840.3578295 is OK
- 10.48550/arXiv.2211.09110 is OK
- 10.48550/arXiv.2306.11698 is OK
- 10.1016/j.simpa.2024.100619 is OK
- 10.48550/arXiv.2206.04615 is OK
- 10.5281/zenodo.12608602 is OK
- 10.48550/arXiv.2401.05561 is OK
- 10.48550/arXiv.1810.01943 is OK
- 10.48550/arXiv.1811.05577 is OK
- 10.1109/TVCG.2019.2934619 is OK
- 10.1145/3340531.3412705 is OK
- 10.48550/arXiv.2407.10853 is OK
- 10.18653/v1/N19-3002 is OK
- 10.48550/arXiv.2311.14126 is OK
- 10.48550/arXiv.1703.06856 is OK
- 10.48550/arXiv.2309.00770 is OK
- 10.18653/v1/2020.findings-emnlp.7 is OK
- 10.1145/3604915.3608860 is OK
- 10.1145/2783258.2783311 is OK
- 10.18653/v1/2021.acl-long.150 is OK
- 10.18653/v1/2022.naacl-main.122 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...
- No DOI given, and none found for title: Fairlearn: Assessing and Improving Fairness of AI ...
- No DOI given, and none found for title: GitHub - tensorflow/fairness-indicators: Tensorflo...
- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

editorialbot · 2025-01-23T15:08:26Z

⚠️ Error preparing paper acceptance. The generated XML metadata file is invalid.

ID ref-huggingface-no-date already defined

dylanbouchard · 2025-01-23T15:10:04Z

^ Duplicate reference, removing now

crvernon · 2025-01-23T15:10:24Z

@dylanbouchard looks like there are two uses of huggingface-no-date in your bib file. Could you please modify one of these?

dylanbouchard · 2025-01-23T15:15:03Z

@crvernon I just removed the duplicate instance of this reference in the bib file. Thank you.

crvernon · 2025-01-23T15:15:31Z

@editorialbot recommend-accept

editorialbot · 2025-01-23T15:15:33Z

Attempting dry run of processing paper acceptance...

editorialbot · 2025-01-23T15:15:51Z

Reference check summary (note 'MISSING' DOIs are suggestions that need verification):

✅ OK DOIs

- 10.48550/arXiv.1804.09301 is OK
- 10.48550/arXiv.1804.06876 is OK
- 10.1162/tacl_a_00240 is OK
- 10.48550/arXiv.2109.03858 is OK
- 10.48550/arXiv.2004.09456 is OK
- 10.48550/arXiv.2010.14534 is OK
- 10.48550/arXiv.2010.00133 is OK
- 10.48550/arXiv.2306.15087 is OK
- 10.48550/arXiv.2106.03521 is OK
- 10.48550/arXiv.2205.12586 is OK
- 10.18653/v1/S18-2005 is OK
- 10.18653/v1/2020.findings-emnlp.301 is OK
- 10.1145/3442188.3445924 is OK
- 10.48550/arXiv.2306.11507 is OK
- 10.18653/v1/2021.naacl-main.191 is OK
- 10.18653/v1/2022.findings-acl.165 is OK
- 10.18653/v1/2020.findings-emnlp.311 is OK
- 10.1145/3576840.3578295 is OK
- 10.48550/arXiv.2211.09110 is OK
- 10.48550/arXiv.2306.11698 is OK
- 10.1016/j.simpa.2024.100619 is OK
- 10.48550/arXiv.2206.04615 is OK
- 10.5281/zenodo.12608602 is OK
- 10.48550/arXiv.2401.05561 is OK
- 10.48550/arXiv.1810.01943 is OK
- 10.48550/arXiv.1811.05577 is OK
- 10.1109/TVCG.2019.2934619 is OK
- 10.1145/3340531.3412705 is OK
- 10.48550/arXiv.2407.10853 is OK
- 10.18653/v1/N19-3002 is OK
- 10.48550/arXiv.2311.14126 is OK
- 10.48550/arXiv.1703.06856 is OK
- 10.48550/arXiv.2309.00770 is OK
- 10.18653/v1/2020.findings-emnlp.7 is OK
- 10.1145/3604915.3608860 is OK
- 10.1145/2783258.2783311 is OK
- 10.18653/v1/2021.acl-long.150 is OK
- 10.18653/v1/2022.naacl-main.122 is OK

🟡 SKIP DOIs

- No DOI given, and none found for title: GitHub - huggingface/evaluate: Evaluate: A library...
- No DOI given, and none found for title: Fairlearn: Assessing and Improving Fairness of AI ...
- No DOI given, and none found for title: GitHub - tensorflow/fairness-indicators: Tensorflo...

❌ MISSING DOIs

- None

❌ INVALID DOIs

- None

editorialbot · 2025-01-23T15:17:46Z

👋 @openjournals/dsais-eics, this paper is ready to be accepted and published.

Check final proof 👉📄 Download article

If the paper PDF and the deposit XML files look good in openjournals/joss-papers#6361, then you can now move forward with accepting the submission by compiling again with the command @editorialbot accept

crvernon · 2025-01-23T15:19:11Z

@editorialbot accept

editorialbot · 2025-01-23T15:19:14Z

Doing it live! Attempting automated processing of paper acceptance...

editorialbot · 2025-01-23T15:26:43Z

Ensure proper citation by uploading a plain text CITATION.cff file to the default branch of your repository.

If using GitHub, a Cite this repository menu will appear in the About section, containing both APA and BibTeX formats. When exported to Zotero using a browser plugin, Zotero will automatically create an entry using the information contained in the .cff file.

You can copy the contents for your CITATION.cff file here:

CITATION.cff

cff-version: "1.2.0"
authors:
- family-names: Bouchard
  given-names: Dylan
  orcid: "https://orcid.org/0009-0004-9233-2324"
- family-names: Chauhan
  given-names: Mohit Singh
  orcid: "https://orcid.org/0000-0002-7817-0427"
- family-names: Skarbrevik
  given-names: David
  orcid: "https://orcid.org/0009-0005-0005-0408"
- family-names: Bajaj
  given-names: Viren
  orcid: "https://orcid.org/0000-0002-9984-1293"
- family-names: Ahmad
  given-names: Zeya
  orcid: "https://orcid.org/0009-0009-1478-2940"
doi: 10.5281/zenodo.14622998
message: If you use this software, please cite our article in the
  Journal of Open Source Software.
preferred-citation:
  authors:
  - family-names: Bouchard
    given-names: Dylan
    orcid: "https://orcid.org/0009-0004-9233-2324"
  - family-names: Chauhan
    given-names: Mohit Singh
    orcid: "https://orcid.org/0000-0002-7817-0427"
  - family-names: Skarbrevik
    given-names: David
    orcid: "https://orcid.org/0009-0005-0005-0408"
  - family-names: Bajaj
    given-names: Viren
    orcid: "https://orcid.org/0000-0002-9984-1293"
  - family-names: Ahmad
    given-names: Zeya
    orcid: "https://orcid.org/0009-0009-1478-2940"
  date-published: 2025-01-23
  doi: 10.21105/joss.07570
  issn: 2475-9066
  issue: 105
  journal: Journal of Open Source Software
  publisher:
    name: Open Journals
  start: 7570
  title: "LangFair: A Python Package for Assessing Bias and Fairness in
    Large Language Model Use Cases"
  type: article
  url: "https://joss.theoj.org/papers/10.21105/joss.07570"
  volume: 10
title: "LangFair: A Python Package for Assessing Bias and Fairness in
  Large Language Model Use Cases"

If the repository is not hosted on GitHub, a .cff file can still be uploaded to set your preferred citation. Users will be able to manually copy and paste the citation.

Find more information on .cff files here and here.

editorialbot · 2025-01-23T15:26:49Z

🐘🐘🐘 👉 Toot for this paper 👈 🐘🐘🐘

editorialbot · 2025-01-23T15:26:50Z

🦋🦋🦋 👉 Bluesky post for this paper 👈 🦋🦋🦋

editorialbot · 2025-01-23T15:26:51Z

🚨🚨🚨 THIS IS NOT A DRILL, YOU HAVE JUST ACCEPTED A PAPER INTO JOSS! 🚨🚨🚨

Here's what you must now do:

Check final PDF and Crossref metadata that was deposited 👉 Creating pull request for 10.21105.joss.07570 joss-papers#6362
Wait five minutes, then verify that the paper DOI resolves https://doi.org/10.21105/joss.07570
If everything looks good, then close this review issue.
Party like you just published a paper! 🎉🌈🦄💃👻🤘

Any issues? Notify your editorial technical team...

crvernon · 2025-01-23T15:36:43Z

🥳 Congratulations on your new publication @dylanbouchard! Many thanks to @xavieryao and @emily-sexton for your time, hard work, and expertise!! JOSS wouldn't be able to function nor succeed without your efforts.

Please consider becoming a reviewer for JOSS if you are not already: https://reviewers.joss.theoj.org/join

editorialbot · 2025-01-23T15:36:45Z

🎉🎉🎉 Congratulations on your paper acceptance! 🎉🎉🎉

If you would like to include a link to your paper from your README use the following

code snippets

Markdown:
[![DOI](https://joss.theoj.org/papers/10.21105/joss.07570/status.svg)](https://doi.org/10.21105/joss.07570)

HTML:
<a style="border-width:0" href="https://doi.org/10.21105/joss.07570">
  <img src="https://joss.theoj.org/papers/10.21105/joss.07570/status.svg" alt="DOI badge" >
</a>

reStructuredText:
.. image:: https://joss.theoj.org/papers/10.21105/joss.07570/status.svg
   :target: https://doi.org/10.21105/joss.07570

This is how it will look in your documentation:

We need your help!

The Journal of Open Source Software is a community-run journal and relies upon volunteer effort. If you'd like to support us please consider doing either one (or both) of the the following:

Volunteering to review for us sometime in the future. You can add your name to the reviewer list here: https://reviewers.joss.theoj.org/join
Making a small donation to support our running costs here: https://numfocus.org/donate-to-open-journals

dylanbouchard · 2025-01-23T15:38:16Z

@crvernon, @xavieryao, @emily-sexton thank you very much for the reviews!

editorialbot added review Track: 5 (DSAIS) Data Science, Artificial Intelligence, and Machine Learning labels Dec 4, 2024

editorialbot assigned crvernon Dec 4, 2024

editorialbot mentioned this issue Dec 4, 2024

[PRE REVIEW]: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases #7558

Closed

This was referenced Dec 31, 2024

Joss paper draft cvs-health/langfair#84

Merged

New FTU check method cvs-health/langfair#85

Merged

editorialbot added the recommend-accept Papers recommended for acceptance in JOSS. label Jan 23, 2025

editorialbot added accepted published Papers published in JOSS labels Jan 23, 2025

crvernon closed this as completed Jan 23, 2025

[REVIEW]: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases #7570

[REVIEW]: LangFair: A Python Package for Assessing Bias and Fairness in Large Language Model Use Cases #7570

Comments

editorialbot commented Dec 4, 2024 • edited Loading

Status

Reviewer instructions & questions

Checklists

editorialbot commented Dec 4, 2024

editorialbot commented Dec 4, 2024

editorialbot commented Dec 4, 2024

editorialbot commented Dec 4, 2024

editorialbot commented Dec 4, 2024

crvernon commented Dec 4, 2024

editorialbot commented Dec 4, 2024

emily-sexton commented Dec 6, 2024 • edited Loading

Review checklist for @emily-sexton

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

xavieryao commented Dec 17, 2024 • edited Loading

Review checklist for @xavieryao

Conflict of interest

Code of Conduct

General checks

Functionality

Documentation

Software paper

emily-sexton commented Dec 21, 2024

dylanbouchard commented Dec 21, 2024

emily-sexton commented Dec 22, 2024

dylanbouchard commented Dec 22, 2024 • edited Loading

xavieryao commented Dec 29, 2024 • edited Loading

crvernon commented Dec 30, 2024

editorialbot commented Dec 30, 2024

crvernon commented Dec 30, 2024

editorialbot commented Dec 30, 2024

dylanbouchard commented Dec 31, 2024

xavieryao commented Jan 1, 2025

dylanbouchard commented Jan 1, 2025 • edited Loading

dylanbouchard commented Jan 1, 2025

editorialbot commented Jan 1, 2025

dylanbouchard commented Jan 2, 2025

crvernon commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

dylanbouchard commented Jan 23, 2025

dylanbouchard commented Jan 23, 2025

crvernon commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

dylanbouchard commented Jan 23, 2025

crvernon commented Jan 23, 2025

dylanbouchard commented Jan 23, 2025

crvernon commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

crvernon commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

editorialbot commented Jan 23, 2025

crvernon commented Jan 23, 2025

editorialbot commented Jan 23, 2025

dylanbouchard commented Jan 23, 2025

editorialbot commented Dec 4, 2024 •

edited

Loading

emily-sexton commented Dec 6, 2024 •

edited

Loading

xavieryao commented Dec 17, 2024 •

edited

Loading

dylanbouchard commented Dec 22, 2024 •

edited

Loading

xavieryao commented Dec 29, 2024 •

edited

Loading

dylanbouchard commented Jan 1, 2025 •

edited

Loading