Manga data collection system

Results at a glance

A fully automated Python and Laravel platform that collects, validates and publishes manga content from four vendors without manual work. It processes thousands of pages daily, keeps a library of more than 800,000 pages up to date, and protects the platform against abusive bots and crawlers.

• 800,000+ pages in the library
• 8,000–20,000 new pages crawled daily
• 30–60 minutes for a daily crawling run
• 4 vendors integrated into one pipeline
• Daily recrawling to detect updated or replaced content
• Automatic validation and publishing without manual intervention
• Automated detection and blocking of abusive bots and crawlers

The challenge

Manga content changes constantly. New chapters appear every day, existing pages get corrected or replaced, and each vendor follows a different content structure. Managing these changes manually does not scale to a library containing hundreds of thousands of pages.

The goal was to build a reliable, unattended system that could collect content from multiple sources, validate every page, publish updates automatically and maintain data quality at scale.

How it works

1. Distributed crawling. Multi-process Python workers use asyncio and concurrent requests to collect 8,000–20,000 pages daily from four vendors. Each daily run completes in approximately 30–60 minutes.
2. Automated validation. Each page is checked for missing images, corrupted assets and metadata inconsistencies before it is published.
3. Automatic publishing. Validated content is stored and made available through the Laravel application without requiring routine manual review.
4. Continuous freshness. Existing pages are recrawled daily to detect changes, refresh outdated content and keep the library current.
5. Storage optimization. Compression and deduplication reduce redundant data and save hundreds of gigabytes of storage overhead.
6. Laravel delivery layer. A Laravel web application serves the structured dataset to readers through the platform's frontend.

Bot and crawler protection

Public-facing content attracts automated traffic, including aggressive scrapers that can consume server resources and repeatedly request the same pages.

The platform monitors request behavior, identifies abusive bots and crawlers, and automatically blocks suspicious traffic while allowing legitimate search engine crawlers. This helps protect server capacity and reduces unauthorized bulk collection of the dataset.

Automatic traffic handling and failover

Traffic is not distributed evenly across the library. A small number of popular manga titles can generate most of the requests, while a slow vendor can affect the reading experience. The platform monitors demand and response times for individual titles and automatically adapts its delivery strategy to keep pages responsive.

1. Intelligent vendor switching. When a manga experiences a surge in traffic, the system distributes requests across the four integrated vendors to balance load and reduce the impact of slow responses.
2. Automatic cloud storage fallback. If responses remain slow after switching vendors, the system automatically downloads the affected content, uploads it to cloud storage and redirects subsequent requests to the stored copy. Popular content can then be served from infrastructure designed to handle high traffic instead of relying on the original vendor for every request.

The entire process runs automatically without requiring manual monitoring or intervention. Readers get more consistent page delivery during traffic spikes, while requests to external vendors remain under control.

User activity and reading history

The Laravel application records user logins and relevant activity while maintaining an individual reading history for each reader.

Readers can resume where they left off, while activity data provides insight into which content is being read and how users interact with the library.

Architecture and technology stack

The system combines asynchronous Python crawling, automated validation, structured database storage and a Laravel frontend into a single content-processing pipeline.

• Python: Crawling, concurrency and automation
• asyncio and requests: Concurrent data collection
• Laravel and PHP: Application backend and frontend
• MySQL: Structured manga and chapter metadata
• Scheduled jobs: Daily crawling and content refreshes
• Storage optimization: Compression and deduplication

My role

I designed and built the complete system, from the Python crawling pipeline to the Laravel application that serves the collected content.

• Designed and implemented multi-vendor crawling workers
• Built the content validation and integrity-checking layer
• Implemented automated storage and publishing workflows
• Developed daily recrawling and content synchronization
• Implemented bot detection and abusive-traffic blocking
• Built user activity tracking and reading history
• Developed the Laravel frontend and database integration