Multi-Agent endless loop when Reviewer follows Guidelines / endless failure in json_repair

[danieldekay]

Describe the bug

I just activated the guidelines in task.json and the reviewer gets stuck on getting an invalid JSON from the other agent, all the time, until after 25 tries it quits and the workflow ends.

The log file seems to ouput the broken JSON -- and it seems to be truncated. Maybe due to the model's output window being full?

{
  "draft": {
    "Historical Examples of Constitutional Crises": "## Historical Examples of Constitutional Crises\n\n### The Nullification Crisis (1832-1833)\n\nThe Nullification Crisis was a significant constitutional crisis in the United States during the early 19th century. It arose from a confrontation between the state of South Carolina and the federal government over the issue of tariffs. South Carolina declared that the federal tariffs of 1828 and 1832 were unconstitutional and therefore null and void within the state. This crisis tested the limits of state versus federal authority and highlighted the tensions between different interpretations of the Constitution. President Andrew Jackson responded by asserting federal supremacy and passing the Force Bill, which authorized the use of military force to ensure compliance with federal laws. This crisis was eventually resolved through a compromise tariff, but it set a precedent for future conflicts over states' rights (Library of Congress, n.d.).\n\n### The Dred Scott Decision (1857)\n\nThe Dred Scott decision by the U.S. Supreme Court in 1857 is often cited as a constitutional crisis because it exacerbated sectional tensions and challenged the concept of judicial review. The Court ruled that African Americans, whether free or enslaved, were not American citizens and therefore had no standing to sue in federal court. Additionally, the decision declared the Missouri Compromise unconstitutional, asserting that Congress had no authority to prohibit slavery in the territories. This ruling intensified the national debate over slavery and contributed to the onset of the Civil War. It also raised questions about the power of the Supreme Court and its role in interpreting the Constitution (National Archives, n.d.).\n\n### The Watergate Scandal (1972-1974)\n\nThe Watergate scandal was a major political scandal in the United States that led to a constitutional crisis involving the executive branch. It began with a break-in at the Democratic National Committee headquarters at the Watergate office complex and the subsequent cover-up by members of the Nixon administration. The crisis escalated when President Richard Nixon refused to release tape recordings that were believed to contain evidence of his involvement in the cover-up. This led to a legal battle over executive privilege and the limits of presidential power. The Supreme Court ultimately ruled in United States v. Nixon that the president had to comply with a subpoena to release the tapes, leading to Nixon's resignation. The Watergate scandal underscored the importance of checks and balances and the rule of law in the American constitutional system (NPR, n.d.).\n\n### The 2000 Presidential Election\n\nThe 2000 U.S. presidential election resulted in a constitutional crisis due to the contested results in Florida. The election between George W. Bush and Al Gore was one of the closest in American history, with the outcome hinging on Florida's 25 electoral votes. Disputes over ballot counting and voter intent led to a series of legal challenges and recounts. The crisis culminated in the Supreme Court's decision in Bush v. Gore, which effectively awarded the presidency to Bush by halting the Florida recount. This case raised questions about the role of the judiciary in electoral disputes and the mechanisms for resolving contested elections under the Constitution (PBS, n.d.).\n\n### The Impeachment of President Donald Trump (2019-2020)\n\nThe impeachment of President Donald Trump marked another constitutional crisis in the United States. The House of Representatives impeached Trump on charges of abuse of power and obstruction of Congress related to his dealings with Ukraine. The crisis centered on the constitutional processes of impeachment and the separation of powers. It highlighted the political divisions in the country and the challenges of holding a sitting president accountable. The Senate ultimately acquitted Trump, but the impeachment proceedings underscored the complexities of the constitutional framework for addressing presidential misconduct (The New York Times, n.d.).\n\n### The Brexit Crisis (2016-2020)\n\nThe United Kingdom's decision to leave the European Union, known as Brexit, led to a constitutional crisis as it exposed the limitations of the UK's unwritten constitution. The crisis involved disputes over the powers of Parliament versus the executive branch, the role of referendums in constitutional change, and the status of devolved governments in Scotland, Wales, and Northern Ireland. Legal challenges, such as the Supreme Court's ruling in Miller v. Secretary of State for Exiting the European Union, addressed the constitutional requirements for triggering Article 50 of the Treaty on European Union. The Brexit crisis highlighted the need for clarity in constitutional processes and the challenges of navigating complex political and legal landscapes (BBC, n.d.).\n\n### The Constitutional Crisis in Poland (2015-Present)\n\nPoland has faced an ongoing constitutional crisis since 2015, involving conflicts between the government and the judiciary. The crisis began with the ruling Law and Justice Party's attempts to exert control over the Constitutional Tribunal and other judicial bodies. These actions have raised concerns about the erosion of judicial independence and the rule of law in Poland. The European Union has intervened, invoking Article 7 of the Treaty on European Union, which allows for sanctions against member states that violate fundamental EU values. The crisis in Poland underscores the challenges of maintaining constitutional democracy in the face of political pressure (European Parliament, n.d.).\n\n### The Venezuelan Constitutional Crisis (2017-Present)\n\nVenezuela has experienced a prolonged constitutional crisis characterized by a power struggle between the executive and legislative branches. The crisis began when President Nicolás Maduro's government attempted to bypass the opposition-controlled National Assembly by creating a Constituent Assembly with the power to rewrite the constitution. This move was widely condemned as unconstitutional and led to widespread protests and international condemnation. The crisis has resulted in political instability, economic collapse, and a humanitarian crisis in Venezuela. It highlights the dangers of undermining constitutional processes and the rule of law (Human Rights Watch, n.d.).\n\n### The Catalonia Independence Referendum (2017)\n\nThe Catalonia independence referendum in Spain led to a constitutional crisis as it challenged the territorial integrity of the Spanish state. The Catalan regional government held a referendum on independence, despite the Spanish Constitutional Court ruling it illegal. The Spanish government responded by invoking Article 155 of the Spanish Constitution, suspending Catalonia's autonomy and imposing direct rule. This crisis raised questions about the right to self-determination, the limits of regional autonomy, and the mechanisms for resolving constitutional disputes in a democratic society (The Guardian, n.d.).\n\n### The Constitutional Crisis in Zimbabwe (2017)\n\nZimbabwe experienced a constitutional crisis in 2017 when the military intervened to remove long-time President Robert Mugabe from power. The crisis arose from a power struggle within the ruling party and concerns about Mugabe's succession plans. The military's actions were widely viewed as a coup, raising questions about the constitutional role of the military and the legitimacy of the transitional government. The crisis highlighted the challenges of constitutional governance in a context of political instability and authoritarian rule (Al Jazeera, n.d.).\n\n### The Constitutional Crisis in Myanmar (2021-Present)\n\nMyanmar's constitutional crisis began with a military coup in February 2021, which overthrew the democratically elected government of Aung San Suu Kyi. The military justified its actions by alleging electoral fraud in the 2020 general election, despite a lack of evidence. The coup has led to widespread protests, a violent crackdown, and a humanitarian crisis. The crisis raises fundamental questions about the role of the military in politics, the protection of democratic institutions, and the international community's response to unconstitutional seizures of power (United Nations, n.d.).\n\n### The Constitutional Crisis in Hong Kong (2019-Present)\n\nHong Kong has faced a constitutional crisis since 2019, driven by tensions between pro-democracy activists and the Chinese government. The crisis began with protests against a proposed extradition bill, which many saw as a threat to Hong Kong's autonomy under the \"one country, two systems\" framework. The Chinese government's subsequent actions, including the imposition of a national security law, have further eroded Hong Kong's autonomy and raised concerns about the future of its democratic institutions. The crisis highlights the challenges of balancing national sovereignty with regional autonomy and the protection of civil liberties (Amnesty International, n.d.).\n\n### The Constitutional Crisis in Turkey (2016-Present)\n\nTurkey has experienced a constitutional crisis since a failed coup attempt in 2016, which led to a crackdown on political dissent and a shift towards authoritarianism. The government of President Recep Tayyip Erdoğan has implemented constitutional changes that concentrate power in the presidency and weaken checks and balances. These changes have raised concerns about the erosion of democratic institutions and the rule of law in Turkey. The crisis underscores the challenges of maintaining constitutional democracy in the face of political instability and authoritarian tendencies (Freedom House, n.d.).\n\n### The Constitutional Crisis in South Africa (2016-Present)\n\nSouth Africa has faced a constitutional crisis involving allegations of corruption and state capture under former President Jacob Zuma. The crisis centered on the influence of the Gupta family over government decisions and the misuse of state resources. The Constitutional Court played a crucial role in addressing these issues, ruling against Zuma in several cases and upholding the principles of accountability and transparency. The crisis highlighted the importance of an independent judiciary and the challenges of combating corruption within a constitutional framework (South African History Online, n.d.).\n\n### The Constitutional Crisis in Brazil (2016-2018)\n\nBrazil experienced a constitutional crisis during the impeachment of President Dilma Rousseff in 2016. The crisis arose from allegations of fiscal mismanagement and corruption, leading to Rousseff's removal from office. The impeachment process was highly contentious and polarized the country, raising questions about the legitimacy of the proceedings and the role of political motivations in constitutional processes. The crisis highlighted the challenges of maintaining democratic governance and the rule of law in a context of political and economic instability (The Economist, n.d.).\n\n### The Constitutional Crisis in Italy (2018)\n\nItaly faced a constitutional crisis in 2018 when President Sergio Mattarella rejected the appointment of a eurosceptic finance minister proposed by the populist coalition government. This decision led to a standoff between the president and the government, raising questions about the limits of presidential powers and the role of the head of state in the formation of government. The crisis was eventually resolved through negotiations, but it underscored the complexities of Italy's constitutional framework and the challenges of balancing political and institutional interests (Reuters, n.d.).\n\n### The Constitutional Crisis in Malaysia (2020)\n\nMalaysia experienced a constitutional crisis in 2020 following the resignation of Prime Minister Mahathir Mohamad and the subsequent political realignment. The crisis involved disputes over the formation of a new government and the role of the king in appointing a prime minister. The crisis highlighted the challenges of navigating constitutional processes in a context of political instability and shifting alliances. It also raised questions about the balance of power between the monarchy and elected representatives in Malaysia's constitutional monarchy (The Straits Times, n.d.).\n\n### The Constitutional Crisis in Lebanon (2019-Present)\n\nLebanon has faced a constitutional crisis since 2019, driven by widespread protests against government corruption and economic mismanagement. The crisis has been exacerbated by political deadlock and the inability to form a stable government. Lebanon's sectarian power-sharing system has been criticized for perpetuating corruption and hindering effective governance. The crisis underscores the challenges of maintaining constitutional democracy in a context of deep-seated political and economic issues (Al Jazeera, n.d.).\n\n### The Constitutional Crisis in Israel (2019-2021)\n\nIsrael experienced a constitutional crisis between 2019 and 2021 due to repeated elections and political deadlock. The crisis arose from the inability to form a stable government and disputes over the role of the judiciary and the separation of powers. The crisis was further complicated by corruption charges against Prime Minister Benjamin Netanyahu and debates over his eligibility to serve while under indictment. The crisis highlighted the challenges of maintaining democratic governance and the rule of law in a context of political polarization and institutional tensions (Haaretz, n.d.).\n\n### The Constitutional Crisis in Thailand (2014-Present)\n\nThailand has faced a constitutional crisis since a military coup in 2014, which led to the suspension of the constitution and the imposition of military rule. The military junta implemented a new constitution in 2017, which critics argue entrenches military influence and undermines democratic governance. The crisis has been marked by ongoing protests and demands for constitutional reform. It highlights the challenges of achieving democratic governance in a context of military intervention and political instability (Human Rights Watch, n.d.).\n\n### The Constitutional Crisis in Egypt (2011-Present)\n\nEgypt has experienced a constitutional crisis since the 2011 revolution that overthrew President Hosni Mubarak. The crisis has involved disputes over the drafting of new constitutions, the role of the military, and the balance of powers between different branches of government. The 2013 military coup that ousted President Mohamed Morsi further complicated the crisis, leading to ongoing political instability and human rights concerns. The crisis underscores the challenges of achieving constitutional democracy in a context of political upheaval and authoritarian tendencies (Amnesty International, n.d.).\n\n### The Constitutional Crisis in Hungary (2010-Present)\n\nHungary has faced a constitutional crisis since 2010, characterized by the erosion of democratic institutions and the rule of law under Prime Minister Viktor Orbán. The crisis has involved changes to the constitution and electoral laws that critics argue consolidate power and weaken checks and balances. The European Union has expressed concerns about Hungary's adherence to democratic principles and the rule of law. The crisis highlights the challenges of maintaining constitutional democracy in the face of authoritarian tendencies and political pressure (European Parliament, n.d.).\n\n### The Constitutional Crisis in Belarus (2020-Present)\n\nBelarus has experienced a constitutional crisis since the disputed presidential election in 2020, which led to widespread protests and international condemnation. The crisis arose from allegations of electoral fraud and the government's crackdown on dissent. The crisis has raised questions about the legitimacy of President Alexander Lukashenko's rule and the future of democratic governance in Belarus. It underscores the challenges of achieving constitutional democracy in a context of authoritarianism and political repression (Human Rights Watch, n.d.).\n\n### The Constitutional Crisis in Russia (1993)\n\nRussia faced a constitutional crisis in 1993 when President Boris Yeltsin dissolved the parliament and called for new elections. The crisis arose from a power struggle between Yeltsin and the parliament, which opposed his economic reforms. The standoff escalated into a violent confrontation, with Yeltsin ordering the military to storm the parliament building. The crisis resulted in the adoption of a new constitution that expanded presidential powers. It highlighted the challenges of transitioning to a democratic system and the risks of concentrating power in the executive branch (BBC, n.d.).\n\n### The Constitutional Crisis in Kenya (2007-2008)\n\nKenya experienced a constitutional crisis following the disputed presidential election in 2007, which led to widespread violence and political instability. The crisis arose from allegations of electoral fraud and ethnic tensions. It was eventually resolved through a power-sharing agreement and constitutional reforms, including the adoption of a new constitution in 2010. The crisis underscored the importance of credible electoral processes and the challenges of achieving national unity in a context of ethnic diversity (The Guardian, n.d.).\n\n### The Constitutional Crisis in Sri Lanka (2018)\n\nSri Lanka faced a constitutional crisis in 2018 when President Maithripala Sirisena dismissed Prime Minister Ranil Wickremesinghe and appointed former President Mahinda Rajapaksa in his place. The crisis led to a standoff between the president and parliament, raising questions about the limits of presidential powers and the role of the judiciary in resolving constitutional disputes. The crisis was eventually resolved through a Supreme Court ruling that reinstated Wickremesinghe. It highlighted the challenges of maintaining constitutional governance in a context of political rivalry (Al Jazeera, n.d.).\n\n### The Constitutional Crisis in Pakistan (2007)\n\nPakistan experienced a constitutional crisis in 2007 when President Pervez Musharraf declared a state of emergency, suspended the constitution, and dismissed the judiciary. The crisis arose from Musharraf's attempts to extend his rule and suppress opposition. It led to widespread protests and international condemnation. The crisis was eventually resolved through elections and the restoration of democratic governance. It underscored the importance of judicial independence and the challenges of maintaining constitutional order in a context of military influence (BBC, n.d.).\n\n### The Constitutional Crisis in Nigeria (2010)\n\nNigeria faced a constitutional crisis in 2010 following the prolonged illness of President Umaru Yar'Adua, which left the country without a clear leader. The crisis arose from the lack of constitutional provisions for the transfer of power in such situations. It was resolved through the invocation of the \"doctrine of necessity,\" which allowed Vice President Goodluck Jonathan to assume the presidency. The crisis highlighted the need for constitutional clarity and the challenges of governance in a context of political uncertainty (The Guardian, n.d.).\n\n### The Constitutional Crisis in South Korea (2016-2017)\n\nSouth Korea experienced a constitutional crisis during the impeachment of President Park Geun-hye in 2016-2017. The crisis arose from allegations of corruption and abuse of power, leading to mass protests and political turmoil. The Constitutional Court upheld the impeachment, resulting in Park's removal from office. The crisis underscored the importance of accountability and the rule of law in a democratic system, as well as the challenges of addressing corruption at the highest levels of government (The New York Times, n.d.).\n\n### The Constitutional Crisis in Greece (2015)\n\nGreece faced a constitutional crisis in 2015 during its financial crisis and negotiations with international creditors. The crisis arose from the government's decision to hold a referendum on austerity measures, which was seen as a challenge to the established constitutional order. The crisis highlighted the tensions between national sovereignty and international obligations, as well as the challenges of maintaining democratic governance in a context of economic hardship (The Guardian, n.d.).\n\n### The Constitutional Crisis in the Philippines (2001)\n\nThe Philippines experienced a constitutional crisis in 2001 when President Joseph Estrada was ousted from office following allegations of corruption and mass protests. The crisis arose from questions about the legitimacy of Estrada's removal and the role of the military and judiciary in the transition of power. The crisis underscored the challenges of maintaining constitutional order and the rule of law in a context of political instability and public discontent (BBC, n.d.).\n\n### The Constitutional Crisis in Indonesia (1998)\n\nIndonesia faced a constitutional crisis in 1998 during the Asian financial crisis, which led to the resignation of President Suharto after 32 years in power. The crisis arose from widespread protests and demands for political reform, leading to a transition to democracy. The crisis highlighted the challenges of achieving constitutional governance in a context of economic instability and authoritarian rule (The New York Times, n.d.).\n\n### The Constitutional Crisis in Argentina (2001)\n\nArgentina experienced a constitutional crisis in 2001 during its economic collapse, which led to

[danieldekay]

@assafelovic Have you considered using JSON output of models? Or does that not work because we are so model agnostic and not all models support this mode?

[danieldekay]

One solution:

SMART_TOKEN_LIMIT=8000´

Prevented the broken JSON effect.

TODO: better handling of the error when a JSON just cannot be repaired by json_repair.

[ElishaKay]

Good point @danieldekay

This will probably be an ongoing issue with correctly converting the JSON response of an LLM from text to JSON.

The current implementation is quite basic.

Have a look here: multi_agents/agents/utils/llms.py

 if response_format == "json":
      try:
          cleaned_json_string = response.strip("```json\n")
          return json.loads(cleaned_json_string)
      except Exception as e:
          print("⚠️ Error in reading JSON, attempting to repair JSON")
          logger.error(
              f"Error in reading JSON, attempting to repair reponse: {response}"
          )
          return json_repair.loads(response)

My vote is a utils function to convert text to JSON which can be imported from the GPTR pip package.

We can define it here: gpt_researcher/utils/llm.py

That will enable us to use it like so:

from gpt_researcher.config.config import Config
from gpt_researcher.utils.llm import create_chat_completion
from gpt_researcher.utils.llm import text_to_json
import asyncio
from dotenv import load_dotenv
load_dotenv()

async def main():
    cfg = Config()

    try:
        generated_text = await create_chat_completion(
            model=cfg.smart_llm_model,
            messages = [{"role": "user", "content": "provide me with a sample json object for the recipe of a delicious pizza"}],
            temperature=0.35,
            llm_provider=cfg.smart_llm_provider,
            stream=True,
            max_tokens=cfg.smart_token_limit,
            llm_kwargs=cfg.llm_kwargs
        )
        print(text_to_json(generated_text))
    except Exception as e:
        print(f"Error in calling LLM: {e}")

# Run the async function
asyncio.run(main())