News Aggregator API

# News Aggregator API (MCP Server) A modular, scalable news aggregation backend built with MCP server conventions and TypeScript. This API provides a unified interface to TheNewsAPI for retrieving current and historical news articles with advanced filtering capabilities. Designed specifically for consumption by AI agents, this API prioritizes structured data and consistent patterns. ## Features - **Unified API**: Consistent interface for accessing news data - **TypeScript**: Fully typed for reliable development - **MCP Architecture**: Scalable, modular design with separation of concerns - **Multiple Endpoints**: Access various types of news content - **Environment Variables**: Secure configuration management - **Error Handling**: Consistent error formats with appropriate status codes - **Filtering**: Support for various filtering parameters - **Caching**: Intelligent caching system with management endpoints - **Interactive Documentation**: Swagger-based API documentation - **AI-Friendly Responses**: Structured data optimized for machine consumption ## Endpoints ### News Endpoints | Endpoint | Description | Example | |----------|-------------|--------| | `/api/news/top` | Get top news headlines | `/api/news/top?categories=business` | | `/api/news/all` | Get all news with advanced search | `/api/news/all?search=technology` | | `/api/news/similar/:uuid` | Get similar articles to a specific one | `/api/news/similar/cc11e3ab-ced0-4a42-9146-e426505e2e67` | | `/api/news/uuid/:uuid` | Get a specific article by UUID | `/api/news/uuid/cc11e3ab-ced0-4a42-9146-e426505e2e67` | | `/api/news/sources` | Get available news sources | `/api/news/sources?language=en` | ### Cache Management Endpoints | Endpoint | Description | Example | |----------|-------------|--------| | `/api/cache/stats` | Get cache statistics | `/api/cache/stats` | | `/api/cache/clear` | Clear all cache | `DELETE /api/cache/clear` | | `/api/cache/clear/:type` | Clear cache by type | `DELETE /api/cache/clear/top` | ### Utility Endpoints | Endpoint | Description | |----------|-------------| | `/health` | Health check endpoint | | `/docs` | Interactive API documentation | | `/docs.json` | OpenAPI specification | | `/examples` | Usage examples | ## Testing This project implements a comprehensive testing strategy with multiple layers of tests to ensure reliability and correctness. ### Test Types 1. **Unit Tests** - Isolated tests for individual components - Mocked dependencies for true unit isolation - Focused on business logic validation 2. **Controller Tests** - Verify controller behavior with mocked services - Test proper request handling and response formatting - Confirm error handling patterns are consistent 3. **Integration Tests** - Test the complete request-response cycle - Verify API endpoints with supertest - Mock external API calls for reliability ### Testing Best Practices This project follows these testing best practices: - **Isolation** - Tests don't depend on each other or external services - **Repeatable** - Tests produce the same results on any environment - **Fast** - Mocking external dependencies keeps tests efficient - **Comprehensive** - Core functionality has high test coverage - **Maintainable** - Tests follow consistent patterns and structure ### Running Tests ```bash # Run all tests npm test # Run tests with coverage npm run test:coverage # Run a specific test file npx jest path/to/test.ts ``` ## Installation ```bash # Clone the repository git clone <repository-url> # Install dependencies npm install # Set up environment variables cp .env.example .env # Edit .env file with your own API key and database URL # Generate Prisma client npx prisma generate # Build the project npm run build # Start the server npm start ``` ## Configuration Create a `.env` file with the following variables: ``` # News Aggregator API Environment Variables # API Configuration NEWS_API_TOKEN=your_api_token_here # Server Configuration PORT=3000 NODE_ENV=development # Database Configuration DATABASE_URL="file:./dev.db" ``` ### Database Configuration The application uses Prisma ORM with SQLite by default. The `DATABASE_URL` environment variable should point to your database file. For production, you can configure it to use PostgreSQL or MySQL. ## Development ```bash # Run in development mode with hot reload npm run dev # Run linting npm run lint # Run tests npm test ``` ## Testing Infrastructure The project includes a comprehensive test suite built with Jest and Supertest, covering API endpoints, database integration, and server connectivity checks. ### Running Tests ```bash # Run all tests npm test # Run only API endpoint tests npm run test:api # Run only database integration tests npm run test:db # Run tests with coverage report npm run test:coverage # Run tests in watch mode (for development) npm run test:watch # Check server connectivity npm run check-server ``` ### Test Structure - **API Endpoint Tests**: Located in `src/__tests__/api/`, these tests verify that all API endpoints return the expected responses, handle errors correctly, and apply appropriate filtering. - **Database Integration Tests**: Located in `src/__tests__/database.test.ts`, these tests verify database connectivity, query execution, and transaction support. - **Server Connectivity Script**: Located in `scripts/check-server.ts`, this script tests connectivity to a running server instance, verifying that critical endpoints are functioning correctly. ### Test Environment Tests use a separate environment configuration specified by setting `NODE_ENV=test`. This ensures tests don't interfere with development or production environments. The setup includes: - Automatic database connection setup and teardown - Cache clearing between tests to ensure isolation - Server startup and shutdown for endpoint testing ## Future Enhancements Todo List ### Priority 1: Foundation Improvements - [x] **Caching System** - Implement in-memory caching - Cache frequently accessed data like top news and sources - Add cache invalidation strategies - Configure TTL (Time-To-Live) for different content types - Add cache management endpoints - [x] **Data Persistence Layer** - Add database integration (SQLite with Prisma) - Create schema for storing enhanced article metadata - Implement repository pattern for data access - Add migration system for schema changes - [x] **API Documentation** - Generate OpenAPI/Swagger specification - Add interactive API documentation - Create usage examples for each endpoint ## API Documentation The API now includes comprehensive documentation using OpenAPI/Swagger: - **Interactive API Documentation**: Available at `/docs` when the server is running - **OpenAPI Specification**: JSON format available at `/docs.json` - **Usage Examples**: Available at `/examples` with code samples in multiple languages ### How to Access the Documentation 1. Start the server with `npm run dev` or `npm start` 2. Open your browser to `http://localhost:3000/docs` for the interactive Swagger UI 3. Browse through the available endpoints, request parameters, and response formats 4. Reference implementation examples at `http://localhost:3000/examples` ### API Documentation Features - Complete endpoint descriptions with parameters and response types - Interactive "Try it out" capability to test endpoints directly - Request/response schema definitions - Code snippets for API consumption - Examples of integrating with the MCP server architecture ### Priority 2: Security & Performance - [ ] **Authentication & User Management** - Implement JWT-based authentication - Create user registration and login endpoints - Add role-based access control - Implement secure password handling - [ ] **Rate Limiting** - Add request throttling middleware - Implement tiered access levels - Create fair usage policies - Add rate limit headers in responses - [ ] **Monitoring & Analytics** - Set up request logging and monitoring - Add performance metrics collection - Create usage dashboards - Implement error tracking and alerting ### Priority 3: Enhanced Functionality - [ ] **Content Processing** - Add text summarization capabilities - Implement entity extraction - Add sentiment analysis - Create topic categorization beyond source categories - Implement duplicate detection and grouping - [ ] **Personalization** - Add user preference tracking - Create personalized endpoints based on reading history - Implement topic following functionality - Add recommendation engine - [ ] **Webhooks & Real-time Updates** - Implement webhook registration - Create event-based notification system - Add SSE (Server-Sent Events) for real-time updates - Implement topic subscription functionality ## Contributing Contributions to the News Aggregator API project are welcome! This project is designed to be a friendly, well-documented codebase for those interested in MCP server architecture, especially with AI consumption in mind. ### How to Contribute 1. Fork the repository 2. Create a new feature branch (`git checkout -b feature/amazing-feature`) 3. Make your changes 4. Run tests to ensure everything works (`npm test`) 5. Commit your changes (`git commit -m 'Add some amazing feature'`) 6. Push to the branch (`git push origin feature/amazing-feature`) 7. Open a Pull Request ### Code Style Please follow the existing code style and patterns in the project. This project uses ESLint for code quality and formatting. ### Security Considerations When contributing, please ensure: - No API keys or credentials are hardcoded - Environment variables are used for configuration - No sensitive data is logged - Input validation is properly implemented ### Testing All new features should include appropriate tests. Please follow the existing testing patterns in the project. - [ ] **Search History & Bookmarks** - Add saved searches functionality - Implement article bookmarking - Create reading history tracking - Add user collections feature ## Contributing 1. Fork the repository 2. Create your feature branch: `git checkout -b feature/my-new-feature` 3. Commit your changes: `git commit -am 'Add some feature'` 4. Push to the branch: `git push origin feature/my-new-feature` 5. Submit a pull request ## License ISC