Add anthropic key

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: [abi]

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,21 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Screenshots of backend AND frontend terminal logs**
+If applicable, add screenshots to help explain your problem.

.github/ISSUE_TEMPLATE/custom.md

@@ -0,0 +1,10 @@
+---
+name: Custom issue template
+about: Describe this issue template's purpose here.
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.gitignore

@@ -11,3 +11,6 @@ backend/backend/*
 # Env vars
 frontend/.env.local
 .env
+
+# Mac files
+.DS_Store

.vscode/settings.json

@@ -0,0 +1,5 @@
+{
+  "python.analysis.typeCheckingMode": "strict",
+  "python.analysis.extraPaths": ["./backend"],
+  "python.autoComplete.extraPaths": ["./backend"]
+}

Evaluation.md

@@ -0,0 +1,19 @@
+## Evaluating models and prompts
+
+Evaluation dataset consists of 16 screenshots. A Python script for running screenshot-to-code on the dataset and a UI for rating outputs is included. With this set up, we can compare and evaluate various models and prompts.
+
+### Running evals
+
+- Input screenshots should be located at `backend/evals_data/inputs` and the outputs will be `backend/evals_data/outputs`. If you want to modify this, modify `EVALS_DIR` in `backend/evals/config.py`. You can download the input screenshot dataset here: TODO.
+- Set a stack and model (`STACK` var, `MODEL` var) in `backend/run_evals.py`
+- Run `OPENAI_API_KEY=sk-... python run_evals.py` - this runs the screenshot-to-code on the input dataset in parallel but it will still take a few minutes to complete.
+- Once the script is done, you can find the outputs in `backend/evals_data/outputs`.
+
+### Rating evals
+
+In order to view and rate the outputs, visit your front-end at `/evals`.
+
+- Rate each output on a scale of 1-4
+- You can also print the page as PDF to share your results with others.
+
+Generally, I run three tests for each model/prompt + stack combo and take the average score out of those tests to evaluate.

README.md

@@ -1,29 +1,48 @@
 # screenshot-to-code
 
-This simple app converts a screenshot to code (HTML/Tailwind CSS, or React or Vue or Bootstrap). It uses GPT-4 Vision to generate the code and DALL-E 3 to generate similar-looking images. You can now also enter a URL to clone a live website!
+A simple tool to convert screenshots, mockups and Figma designs into clean, functional code using AI. **Now supporting GPT-4O!**
 
 https://github.com/abi/screenshot-to-code/assets/23818/6cebadae-2fe3-4986-ac6a-8fb9db030045
 
-See the [Examples](#examples) section below for more demos.
+Supported stacks:
 
-## 🚀 Try It Out!
+- HTML + Tailwind
+- React + Tailwind
+- Vue + Tailwind
+- Bootstrap
+- Ionic + Tailwind
+- SVG
 
-🆕 [Try it here](https://screenshottocode.com) (bring your own OpenAI key - **your key must have access to GPT-4 Vision. See [FAQ](#%EF%B8%8F-faqs) section below for details**). Or see [Getting Started](#-getting-started) below for local install instructions.
+Supported AI models:
 
-## 🌟 Recent Updates
+- GPT-4O - Best model!
+- GPT-4 Turbo (Apr 2024)
+- GPT-4 Vision (Nov 2023)
+- Claude 3 Sonnet
+- DALL-E 3 for image generation
 
-- Nov 28 - 🔥 🔥 🔥 Get output code in React or Bootstrap or TailwindCSS
-- Nov 23 - Send in a screenshot of the current replicated version (sometimes improves quality of subsequent generations)
-- Nov 21 - Edit code in the code editor and preview changes live thanks to [@clean99](https://github.com/clean99)
-- Nov 20 - Paste in a URL to screenshot and clone (requires [ScreenshotOne free API key](https://screenshotone.com?via=screenshot-to-code))
-- Nov 19 - Support for dark/light code editor theme - thanks [@kachbit](https://github.com/kachbit)
-- Nov 16 - Added a setting to disable DALL-E image generation if you don't need that
-- Nov 16 - View code directly within the app
-- Nov 15 - You can now instruct the AI to update the code as you wish. It is helpful if the AI messed up some styles or missed a section.
+See the [Examples](#-examples) section below for more demos.
+
+We also just added experimental support for taking a video/screen recording of a website in action and turning that into a functional prototype. 
+
+![google in app quick 3](https://github.com/abi/screenshot-to-code/assets/23818/8758ffa4-9483-4b9b-bb66-abd6d1594c33)
+
+[Learn more about video here](https://github.com/abi/screenshot-to-code/wiki/Screen-Recording-to-Code).
+
+[Follow me on Twitter for updates](https://twitter.com/_abi_).
+
+## Sponsors
+
+<a href="https://konghq.com/products/kong-konnect/register?utm_medium=referral&utm_source=github&utm_campaign=platform&utm_content=screenshot-to-code" target="_blank" title="Kong - powering the API world"><img src="https://picoapps.xyz/s2c-sponsors/Kong-GitHub-240x100.png"></a>
+
+
+## 🚀 Try It Out without no install
+
+[Try it live on the hosted version (paid)](https://screenshottocode.com).
 
 ## 🛠 Getting Started
 
-The app has a React/Vite frontend and a FastAPI backend. You will need an OpenAI API key with access to the GPT-4 Vision API.
+The app has a React/Vite frontend and a FastAPI backend. You will need an OpenAI API key with access to the GPT-4 Vision API or an Anthropic key if you want to use Claude Sonnet, or for experimental video support.
 
 Run the backend (I use Poetry for package management - `pip install poetry` if you don't have it):
 
@@ -35,6 +54,8 @@ poetry shell
 poetry run uvicorn main:app --reload --port 7001
 ```
 
+If you want to use Anthropic, add the `ANTHROPIC_API_KEY` to `backend/.env` with your API key from Anthropic.
+
 Run the frontend:
 
 ```bash
@@ -68,6 +89,9 @@ The app will be up and running at http://localhost:5173. Note that you can't dev
 
 - **I'm running into an error when setting up the backend. How can I fix it?** [Try this](https://github.com/abi/screenshot-to-code/issues/3#issuecomment-1814777959). If that still doesn't work, open an issue.
 - **How do I get an OpenAI API key?** See https://github.com/abi/screenshot-to-code/blob/main/Troubleshooting.md
+- **How can I configure an OpenAI proxy?** - If you're not able to access the OpenAI API directly (due to e.g. country restrictions), you can try a VPN or you can configure the OpenAI base URL to use a proxy: Set OPENAI_BASE_URL in the `backend/.env` or directly in the UI in the settings dialog. Make sure the URL has "v1" in the path so it should look like this:  `https://xxx.xxxxx.xxx/v1`
+- **How can I update the backend host that my front-end connects to?** - Configure VITE_HTTP_BACKEND_URL and VITE_WS_BACKEND_URL in front/.env.local For example, set VITE_HTTP_BACKEND_URL=http://124.10.20.1:7001
+- **Seeing UTF-8 errors when running the backend?** - On windows, open the .env file with notepad++, then go to Encoding and select UTF-8. 
 - **How can I provide feedback?** For feedback, feature requests and bug reports, open an issue or ping me on [Twitter](https://twitter.com/_abi_).
 
 ## 📚 Examples
@@ -88,6 +112,6 @@ https://github.com/abi/screenshot-to-code/assets/23818/3fec0f77-44e8-4fb3-a769-a
 
 ## 🌍 Hosted Version
 
-🆕 [Try it here](https://screenshottocode.com) (bring your own OpenAI key - **your key must have access to GPT-4 Vision. See [FAQ](#%EF%B8%8F-faqs) section for details**). Or see [Getting Started](#-getting-started) for local install instructions.
+🆕 [Try it here (paid)](https://screenshottocode.com). Or see [Getting Started](#-getting-started) for local install instructions to use with your own API keys.
 
 [!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://www.buymeacoffee.com/abiraja)

Troubleshooting.md

@@ -1,16 +1,22 @@
-### Getting an OpenAI API key
+### Getting an OpenAI API key with GPT4-Vision model access
 
 You don't need a ChatGPT Pro account. Screenshot to code uses API keys from your OpenAI developer account. In order to get access to the GPT4 Vision model, log into your OpenAI account and then, follow these instructions:
 
 1. Open [OpenAI Dashboard](https://platform.openai.com/)
 1. Go to Settings > Billing
 1. Click at the Add payment details
-<img width="1030" alt="285636868-c80deb92-ab47-45cd-988f-deee67fbd44d" src="https://github.com/abi/screenshot-to-code/assets/23818/4e0f4b77-9578-4f9a-803c-c12b1502f3d7">
-4. You have to buy some credits. The minimum is $5.
+<img width="900" alt="285636868-c80deb92-ab47-45cd-988f-deee67fbd44d" src="https://github.com/abi/screenshot-to-code/assets/23818/4e0f4b77-9578-4f9a-803c-c12b1502f3d7">
 
+4. You have to buy some credits. The minimum is $5.
 5. Go to Settings > Limits and check at the bottom of the page, your current tier has to be "Tier 1" to have GPT4 access
-<img width="785" alt="285636973-da38bd4d-8a78-4904-8027-ca67d729b933" src="https://github.com/abi/screenshot-to-code/assets/23818/8d07cd84-0cf9-4f88-bc00-80eba492eadf">
+<img width="900" alt="285636973-da38bd4d-8a78-4904-8027-ca67d729b933" src="https://github.com/abi/screenshot-to-code/assets/23818/8d07cd84-0cf9-4f88-bc00-80eba492eadf">
+
+6. Navigate to OpenAI [api keys](https://platform.openai.com/api-keys) page and create and copy a new secret key.
+7. Go to Screenshot to code and paste it in the Settings dialog under OpenAI key (gear icon). Your key is only stored in your browser. Never stored on our servers.
+
+## Still not working?
 
-Some users have also reported that it can take upto 30 minutes after your credit purchase for the GPT4 vision model to be activated.
+- Some users have also reported that it can take upto 30 minutes after your credit purchase for the GPT4 vision model to be activated.
+- You need to add credits to your account AND set it to renew when credits run out in order to be upgraded to Tier 1. Make sure your "Settings > Limits" page shows that you are at Tier 1.
 
-If you've followed these steps, and it still doesn't work, feel free to open a Github issue.
+If you've followed these steps, and it still doesn't work, feel free to open a Github issue. We only provide support for the open source version since we don't have debugging logs on the hosted version. If you're looking to use the hosted version, we recommend getting a paid subscription on screenshottocode.com

backend/.gitignore

@@ -150,3 +150,11 @@ cython_debug/
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
 #.idea/
+
+
+# Temporary eval output
+evals_data
+
+
+# Temporary video evals (Remove before merge)
+video_evals

backend/.pre-commit-config.yaml

@@ -0,0 +1,25 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v3.2.0
+    hooks:
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+  - repo: local
+    hooks:
+      - id: poetry-pytest
+        name: Run pytest with Poetry
+        entry: poetry run --directory backend pytest
+        language: system
+        pass_filenames: false
+        always_run: true
+        files: ^backend/
+      # - id: poetry-pyright
+      #   name: Run pyright with Poetry
+      #   entry: poetry run --directory backend pyright
+      #   language: system
+      #   pass_filenames: false
+      #   always_run: true
+      #   files: ^backend/

backend/README.md

@@ -1,3 +1,7 @@
-Run tests
+# Run the type checker
 
-pytest test_prompts.py
+poetry run pyright
+
+# Run tests
+
+poetry run pytest

backend/config.py

@@ -0,0 +1,16 @@
+# Useful for debugging purposes when you don't want to waste GPT4-Vision credits
+# Setting to True will stream a mock response instead of calling the OpenAI API
+# TODO: Should only be set to true when value is 'True', not any abitrary truthy value
+import os
+
+ANTHROPIC_API_KEY = os.environ.get("ANTHROPIC_API_KEY", None)
+
+# Debugging-related
+
+SHOULD_MOCK_AI_RESPONSE = bool(os.environ.get("MOCK", False))
+IS_DEBUG_ENABLED = bool(os.environ.get("IS_DEBUG_ENABLED", False))
+DEBUG_DIR = os.environ.get("DEBUG_DIR", "")
+
+# Set to True when running in production (on the hosted version)
+# Used as a feature flag to enable or disable certain features
+IS_PROD = os.environ.get("IS_PROD", False)

backend/custom_types.py

@@ -0,0 +1,7 @@
+from typing import Literal
+
+
+InputMode = Literal[
+    "image",
+    "video",
+]

backend/debug/DebugFileWriter.py

@@ -0,0 +1,30 @@
+import os
+import logging
+import uuid
+
+from config import DEBUG_DIR, IS_DEBUG_ENABLED
+
+
+class DebugFileWriter:
+    def __init__(self):
+        if not IS_DEBUG_ENABLED:
+            return
+
+        try:
+            self.debug_artifacts_path = os.path.expanduser(
+                f"{DEBUG_DIR}/{str(uuid.uuid4())}"
+            )
+            os.makedirs(self.debug_artifacts_path, exist_ok=True)
+            print(f"Debugging artifacts will be stored in: {self.debug_artifacts_path}")
+        except:
+            logging.error("Failed to create debug directory")
+
+    def write_to_file(self, filename: str, content: str) -> None:
+        try:
+            with open(os.path.join(self.debug_artifacts_path, filename), "w") as file:
+                file.write(content)
+        except Exception as e:
+            logging.error(f"Failed to write to file: {e}")
+
+    def extract_html_content(self, text: str) -> str:
+        return str(text.split("<html>")[-1].rsplit("</html>", 1)[0] + "</html>")

backend/evals/config.py

@@ -0,0 +1 @@
+EVALS_DIR = "./evals_data"

backend/evals/core.py

@@ -0,0 +1,39 @@
+import os
+from config import ANTHROPIC_API_KEY
+
+from llm import Llm, stream_claude_response, stream_openai_response
+from prompts import assemble_prompt
+from prompts.types import Stack
+
+
+async def generate_code_core(image_url: str, stack: Stack, model: Llm) -> str:
+    prompt_messages = assemble_prompt(image_url, stack)
+    openai_api_key = os.environ.get("OPENAI_API_KEY")
+    anthropic_api_key = ANTHROPIC_API_KEY
+    openai_base_url = None
+
+    async def process_chunk(content: str):
+        pass
+
+    if model == Llm.CLAUDE_3_SONNET:
+        if not anthropic_api_key:
+            raise Exception("Anthropic API key not found")
+
+        completion = await stream_claude_response(
+            prompt_messages,
+            api_key=anthropic_api_key,
+            callback=lambda x: process_chunk(x),
+        )
+    else:
+        if not openai_api_key:
+            raise Exception("OpenAI API key not found")
+
+        completion = await stream_openai_response(
+            prompt_messages,
+            api_key=openai_api_key,
+            base_url=openai_base_url,
+            callback=lambda x: process_chunk(x),
+            model=model,
+        )
+
+    return completion

backend/evals/utils.py

@@ -0,0 +1,7 @@
+import base64
+
+
+async def image_to_data_url(filepath: str):
+    with open(filepath, "rb") as image_file:
+        encoded_string = base64.b64encode(image_file.read()).decode()
+    return f"data:image/png;base64,{encoded_string}"