ROADMAP.md

Imageboard Grabber - Further Development & Roadmap

This document outlines a roadmap and future development plan for the Imageboard Grabber application on Hugging Face Spaces.

---

🚀 1. Feature Enhancements

1.1 Support for Authenticated APIs (API Keys & Username)

• Goal: Allow users to safely search and download from private or rate-limited sites (like Danbooru premium tiers, e621 personal accounts).
• Implementation:
  • Add credential fields (API Key / Password Hash, Username) to the Add Site dialog.
  • Securely transmit credentials to the server using HTTP headers or request bodies.
  • Store credentials locally in the browser's localStorage or session-only cookies so they are never stored globally on the server.

1.2 Persistent Site Configuration

• Goal: Avoid losing newly added schemas when refreshing or clearing session storage.
• Implementation:
  • Sync the list of added custom sites with IndexedDB or localStorage.
  • Provide an Export/Import Sites button to allow users to backup their custom configurations as JSON.

1.3 Advanced Query Builder

• Goal: Help non-technical users build complex booru search queries.
• Implementation:
  • Add UI controls for searching by rating (Safe, Questionable, Explicit), score range, upload date, or file extension.
  • Autocomplete / tag suggestion box by querying the active site's tag API endpoint (e.g., /tags.json?search[name_matches]=*).

1.4 Enhanced Lightbox & Media Viewer

• Goal: Support video playback and metadata exploration.
• Implementation:
  • Auto-detect video extensions (.webm, .mp4) in the Lightbox and render a native HTML <video> player with volume and speed controls.
  • Show side-panel info: Artist, Character, Copyright, and Meta tags. Click a tag to search.

---

🛠️ 2. Architectural & Performance Improvements

2.1 Concurrent / Streamed Server Downloads

• Goal: Speed up ZIP compilation and handle massive file downloads without causing OOM (Out Of Memory) crashes.
• Implementation:
  • Implement streaming ZIP using fflate on the server so bytes are sent back to the client immediately as they download, rather than buffering entire gigabytes in server memory.
  • Limit concurrent download workers per request to avoid IP bans from image hosters.

2.2 Client-Side Image Caching

• Goal: Reduce bandwidth usage and accelerate browsing.
• Implementation:
  • Register a Service Worker to cache thumbnails and UI assets.
  • Use Cache-Control headers for downloaded image URLs.

---

🧪 3. Quality Assurance & Automated Testing

3.1 End-to-End Tests with Playwright

• Goal: Ensure code modifications do not break key features like site additions or tag searches.
• Implementation:
  • Add E2E test suites inside a tests/ directory.
  • Mock API responses from e621/Danbooru to test search, grid rendering, and selection UI without hitting actual boorus.

3.2 Unit and API Integration Tests

• Goal: Validate parsing logic, schema validation, and URL building helpers.
• Implementation:
  • Write Vitest suites to verify mapping functions in src/lib/grabber/sites.ts and zip streaming endpoints.

---

🌐 4. Deployment & Infrastructure Optimization

4.1 Hugging Face Storage Integration

• Goal: Retain user session data or cache assets on persistent storage.
• Implementation:
  • Connect a Hugging Face Dataset or persistent Space storage to backup logs or temporary configuration files if requested.