SmartScrapAI - Autonomous News Aggregation System

Overview

SmartScrapAI is an intelligent news aggregation system that automatically collects, processes, and publishes news articles. It uses AI to search, summarize, and create comprehensive news roundups on specific topics.

Features

• 🔍 Smart News Search: Intelligent search across multiple news sources
• 📝 AI-Powered Summarization: Automatic article summarization using BART
• 🌐 Multi-language Support: Translates content into multiple languages
• 🖼️ AI Image Generation: Creates relevant images for articles
• 📊 Batch Processing: Efficient handling of multiple articles
• 🔄 Real-time Processing: Immediate article fetching and processing
• 📱 Responsive UI: Modern and user-friendly interface
• 🚀 One-Click Publishing: Direct integration with publishing platforms

Contributors

• Gaurav Singh - Lead Developer
• Krishnamurthy - Core Developer

Tech Stack

• Backend: Python, Flask
• Frontend: HTML, CSS, JavaScript, jQuery
• AI/ML:
  • BART for summarization
  • Google Translator for multi-language support
  • Pollinations.ai for image generation
• External Services:
  • DuckDuckGo for news search
  • Dev.to for publishing

Installation

1. Clone the repository:

git clone https://github.com/gaurav4288/SmartScrapAI.git
cd SmartScrapAI

2. Create and activate a virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies:

pip install -r requirements.txt

4. Set up environment variables:

export HASHNODE_API_TOKEN=your_token_here
export PUBLICATION_ID=your_publication_id

Usage

1. Start the Flask server:

python manage.py runserver

2. Open your browser and navigate to:

http://localhost:8000

3. Enter your search query, select language preferences, and click search.

4. Select articles to include in your roundup and click "Publish" to create a new blog post.

Configuration

The application can be configured through environment variables:

• Dev.to API Key: Get this from https://dev.to/settings/account
• SECRET_KEY: Flask application secret key
• PORT: Server port (default: 8000)

Contributing

1. Fork the repository
2. Create your feature branch (git checkout -b feature/AmazingFeature)
3. Commit your changes (git commit -m 'Add some AmazingFeature')
4. Push to the branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

This project is licensed under the MIT License - see the LICENSE file for details.

Acknowledgments

• Newspaper3k for article extraction
• Transformers for AI-powered summarization
• DuckDuckGo Search for news search
• Flask for the web framework
• Dev.to for publishing integration

Contact

For support or inquiries:

• Gaurav Singh

  • Email: [email protected]
  • LinkedIn: Gaurav Singh

• Krishnamurthy

  • Email: [email protected]
  • LinkedIn: Krishna Murthi

For general support, please open an issue in the GitHub repository.