Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc/revised overview #351

Merged
merged 13 commits into from
Apr 17, 2023
Binary file modified docs/img/overview.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 57 additions & 7 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,71 @@
# Autonomous Empirical Research
# Automated Research Assistant

AutoRA (Autonomous Research Assistant) is an open-source AI-based system for automating each aspect of empirical research in the behavioral sciences, from the construction of a scientific hypothesis to conducting novel experiments.
<b>[AutoRA](https://pypi.org/project/autora/)</b> (<b>Au</b>tomated <b>R</b>esearch <b>A</b>ssistant) is an open-source framework for
automating multiple stages of the empirical research process, including model discovery, experimental design, data collection, and documentation for open science.

![Overview](img/overview.png)
![Autonomous Empirical Research Paradigm](img/overview.png)

AutoRA implements the <b>Autonomous Empirical Research Paradigm</b>, which involves a dynamic interplay
between two artificial agents. The first agent, a theorist, is primarily responsible for constructing
computational models by relying on existing data to link experimental conditions to dependent measures.
The second agent, an experimentalist, is tasked with designing follow-up
experiments that can refine and validate the models generated by the theorist. Together, these agents
implement an automated scientific discovery process. To enable closed-loop empirical research, AutoRA
interfaces with platforms for automated data collection, such as Prolific or Amazon Mechanical Turk,
which enable the efficient acquisition of behavioral data from human participants. Finally, AutoRA
is designed to support the automated documentation and dissemination of steps in the empirical research process.

AutoRA was initially intended for accelerating research in the behavioral and brain sciences.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
AutoRA was initially intended for accelerating research in the behavioral and brain sciences.
AutoRA was initially intended to accelerate research in the behavioral and brain sciences.

However, AutoRA is designed as a general framework that enables automation of the research processes in
other empirical sciences, such as material science or physics.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
other empirical sciences, such as material science or physics.
other empirical sciences, such as materials science and physics.


## Features

AutoRA consists of different modules that can be used independently or in combination, such as:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
AutoRA consists of different modules that can be used independently or in combination, such as:
AutoRA consists of different modules that can be used independently or in combination, including the following:


- <b>Automated theorists</b> that support the discovery of formal scientific models from data
- <b>Automated experimentalists</b> that support the design of follow-up experiments
- <b>Interfaces for automated data collection</b>, e.g., for behavioral experiments via Prolific or Amazon Mechanical Turk
- <b>Workflow logic</b> for defining interactions between different components of the research process
- <b>Interfaces for automated documentation</b> of the research process

## Usages

AutoRA can be used for a variety of research purposes in empirical sciences, such as psychology,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
AutoRA can be used for a variety of research purposes in empirical sciences, such as psychology,
There are many use cases for AutoRA across the empirical sciences, such as psychology,

neuroscience, economics, physics, or material science. Usages include:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
neuroscience, economics, physics, or material science. Usages include:
neuroscience, economics, physics, or materials science. These use cases include:


- <b>Equation discovery</b> from empirical data
- <b>Experimental design</b> for follow-up experiments
- <b>Research documentation and dissemination</b>
- <b>Closed-loop empirical research</b>
- <b>Computational analyses of the scientific process</b> (metascience, computational philosophy of science)

## Motivation

The integration of behavioral phenomena into computational models of human behavior is a fundamental staple of empirical research in the behavioral sciences, such as cognitive neuroscience and psychology. Yet, researchers are beginning to accumulate increasing amounts of data without having the temporal, computational, or monetary resources to integrate these data into scientific theories, and/or to test resulting theories in follow-up experiments. AutoRA seeks to accelerate the empirical research process, by incorporating existing machine learning techniques into a closed-loop system for the generation, estimation, and experimental validation of scientific models. In addition, AutoRA seeks to facilitate open science, by automating the documentation and dissemination of each step in the empirical research process.
Various empirical sciences are beset by a replication crisis,
which can be attributed to inadequately precise hypotheses, lack of transparency
in research procedures, and insufficient rigor in testing findings. These limitations
are the result of three primary bottlenecks—a lack of formal modeling, the
demanding requirements of open science, and a shortage of resources to reproduce
individual studies. Empirical scientists face difficulties in formalizing their
theories, find it arduous to document their research activities, and often lack
time and funds to conduct follow-up experiments to test and revise their
hypotheses. These limitations impede scientific progress and hinder
the development of new knowledge. We seek to overcome these limitations by providing
a tool for the generation, estimation, and empirical testing of scientific models.
It is our hope that AutoRA will help accelerate scientific discovery by overcoming
these limitations and promoting greater transparency and rigor in empirical research.

## Overview
## Pointers

Autonomous empirical research consists of three core components, each automating a different step in the empirical research process: (1) an autonomous theorist that constructs computational models, linking experiment conditions to dependent measures; (2) an autonomous experimentalist that derives predictions from these models, and designs novel experiments to test them; as well as (3) an experiment environment for automated data collection. The (competitive) interaction between the autonomous theorist, whose objective is to build models based on available data, and the autonomous experimentalist, whose objective is to invalidate these models by collecting new data, implements a form of adversarial curiosity.
- [AutoRA Pip Package](https://pypi.org/project/autora/)
- [GitHub Repository](https://github.com/AutoResearch/autora)
- [Autonomous Empirical Research Group](http://www.empiricalresearch.ai)

## About

This project is in active development by the [Autonomous Empirical Research Group](https://musslick.github.io/AER_website/Research.html), led by [Sebastian Musslick](https://smusslick.com), in collaboration with the [Center for Computation and Visualization at Brown University](https://ccv.brown.edu).

This research program is supported by Schmidt Science Fellows, in partnership with the Rhodes Trust, as well as the Carney BRAINSTORM program at Brown University.
The development of this package is supported by Schmidt Science Fellows, in partnership with the Rhodes Trust, as well as the Carney BRAINSTORM program at Brown University.


82 changes: 45 additions & 37 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -59,42 +59,50 @@ extra_javascript:

nav:
- Introduction: 'index.md'
- Experimentalists:
- Overview: 'experimentalists/overview.md'
- Pipeline: 'pipeline/Experimentalist Pipeline Examples.ipynb'
- Synthetic Experiments:
- Inventory: 'synthetic/inventory.ipynb'
- Theorists:
- Overview: 'theorist/overview.md'
- DARTS:
- Introduction: 'theorist/darts/introduction.md'
- How it works: 'theorist/darts/how_it_works.md'
- Meta parameters: 'theorist/darts/meta_parameters.md'
- Search space: 'theorist/darts/search_space.md'
- Tutorials:
- Automated Theorist
- Automated Experimentalist
- Closed-Loop Discovery
- User Guide:
- Theorists:
- Overview: 'theorist/overview.md'
- DARTS:
- Introduction: 'theorist/darts/introduction.md'
- How it works: 'theorist/darts/how_it_works.md'
- Meta parameters: 'theorist/darts/meta_parameters.md'
- Search space: 'theorist/darts/search_space.md'
- Examples:
- Basic Usage: 'theorist/darts/example.ipynb'
- Weber-Fechner Law: 'theorist/darts/weber.ipynb'
- BMS:
- Introduction: 'theorist/bms/introduction.md'
- How it works: 'theorist/bms/how_it_works.md'
- Meta parameters: 'theorist/bms/meta_parameters.md'
- Search space: 'theorist/bms/search_space.ipynb'
- Examples:
- Basic Usage: 'theorist/bms/example.ipynb'
- Weber-Fechner Law: 'theorist/bms/weber.ipynb'
- BSR:
- Introduction: 'theorist/bsr/introduction.md'
- How it works: 'theorist/bsr/how_it_works.md'
- Meta parameters: 'theorist/bsr/meta_parameters.md'
- Search space: 'theorist/bsr/search_space.md'
- Example: 'theorist/bsr/bsr.ipynb'
- Experimentalists:
- Overview: 'experimentalists/overview.md'
- Pipeline: 'pipeline/Experimentalist Pipeline Examples.ipynb'
- Synthetic Experiments:
- Inventory: 'synthetic/inventory.ipynb'
- Cycle:
- Scoring:
- Logistic Regression: 'cycle/cycle_scoring.ipynb'
- BMS: 'cycle/cycle_scoring_bms.ipynb'
- Plotting: 'cycle/cycle_results_plots.ipynb'
- Examples:
- Basic Usage: 'theorist/darts/example.ipynb'
- Weber-Fechner Law: 'theorist/darts/weber.ipynb'
- BMS:
- Introduction: 'theorist/bms/introduction.md'
- How it works: 'theorist/bms/how_it_works.md'
- Meta parameters: 'theorist/bms/meta_parameters.md'
- Search space: 'theorist/bms/search_space.ipynb'
- Examples:
- Basic Usage: 'theorist/bms/example.ipynb'
- Weber-Fechner Law: 'theorist/bms/weber.ipynb'
- BSR:
- Introduction: 'theorist/bsr/introduction.md'
- How it works: 'theorist/bsr/how_it_works.md'
- Meta parameters: 'theorist/bsr/meta_parameters.md'
- Search space: 'theorist/bsr/search_space.md'
- Example: 'theorist/bsr/bsr.ipynb'
- Cycle:
- Scoring:
- Logistic Regression: 'cycle/cycle_scoring.ipynb'
- BMS: 'cycle/cycle_scoring_bms.ipynb'
- Plotting: 'cycle/cycle_results_plots.ipynb'
- Examples:
- DARTS & BMS: 'cycle/simple_cycle_bms_darts.ipynb'
- BMS & Poppernet: 'cycle/simple_cycle_bms_model_poppernet.ipynb'
- Uncertainty Experimentalist: 'cycle/simple_cycle_uncertainty_experimentalist.ipynb'
- DARTS & BMS: 'cycle/simple_cycle_bms_darts.ipynb'
- BMS & Poppernet: 'cycle/simple_cycle_bms_model_poppernet.ipynb'
- Uncertainty Experimentalist: 'cycle/simple_cycle_uncertainty_experimentalist.ipynb'
- Contributor Guide:
- Module Contributions
- Core Contributions
- Code Reference: reference/autora/