MagicXML is a FastAPI application designed to transform and process XML data streams from URLs effortlessly. With a focus on efficiency and scalability, MagicXML parses large XML files in chunks, extracting valuable information and saving it into well-structured CSV files. This tool is perfect for anyone needing to handle substantial XML data sources, offering an intuitive web interface and robust API endpoints for seamless integration.
- 📦 Chunked Data Fetching: Fetch XML data in chunks to handle large files without overloading memory.
- 🔍 XML Parsing: Efficiently parse XML data and extract relevant details.
- 📄 CSV Export: Save parsed data into neatly organized CSV files.
- 📥 File Download: Provide easy access to the generated CSV files via a download endpoint.
- 🌐 Web Interface: Serve static files and templates for an intuitive user experience.
- Python 3.8+
- FastAPI: A modern, fast (high-performance), web framework for building APIs with Python 3.6+.
- aiohttp: An asynchronous HTTP client/server framework.
- aiofiles: A library for handling local file operations asynchronously.
- Jinja2: A templating engine for Python.
To use the API, you can send a POST request to the /process_link endpoint with the necessary parameters. Below is an example using curl:
curl -X 'POST' \
'https://solarxml.replit.app//process_link' \
-H 'Content-Type: application/json' \
-d '{"link_url": "YOUR_XML_URL", "preset_id": "id=1234"}' \
-o process_response.jsonReplace YOUR_XML_URL with the actual URL of the XML data you want to process. This request will save the response in a file named process_response.json.
-
Clone the repository:
git clone https://github.com/Solrikk/MagicXML.git cd MagicXML -
Create and activate a virtual environment:
python -m venv venv source venv/bin/activate # On Windows use `venv\Scripts\activate`
-
Install the dependencies:
pip install -r requirements.txt
-
Run the FastAPI application:
uvicorn main:app --reload
-
Open your browser and go to
http://127.0.0.1:8000to access the web interface.
- Description: Renders the index page.
- Response: HTML page.
- Description: Processes the given link to fetch, parse, and save XML data into a CSV file.
- Request Body:
link_url(str): The URL to fetch the XML data from.preset_id(str, optional): An optional preset ID.
- Response: JSON containing the URL of the generated CSV file and the preset ID.
- Description: Downloads the specified CSV file.
- Path Parameter:
filename(str): The name of the CSV file to download.
- Response: The requested CSV file.
main.py: The main application file.templates/: Directory containing HTML templates.static/: Directory containing static files (CSS, JS, images).data_files/: Directory where the generated CSV files are saved.
- Imports: Various libraries and modules for HTTP handling, asynchronous operations, file handling, and XML parsing.
- FastAPI Setup: Initializes the FastAPI app, sets up Jinja2 templates, and mounts the static files directory.
- Data Models: Defines the
LinkDatamodel using Pydantic for request validation. - Utility Functions: Includes functions for removing unwanted HTML tags from descriptions and fetching URL data in chunks.
- Processing Functions: Contains asynchronous functions to process XML data, parse it, and save it into CSV files.
- API Endpoints: Defines endpoints for the root page, processing links, and downloading CSV files.
remove_unwanted_tags(description): Removes HTML tags from a given description string.fetch_url_in_chunks(link_url, chunk_size=1024): Fetches data from a URL in chunks and yields the data.process_offer(offer_elem, build_category_path): Processes individual XML elements to extract offer data.process_link_stream(link_url, chunk_size=1024): Processes the XML data stream from the URL, parses it, and writes it to a CSV file.