Data Importer | Projects

This Django project features a robust data importer tool designed to handle large file imports efficiently. Leveraging Django's custom management commands, MongoDB integration, and asynchronous task processing with Celery and Redis, this project ensures seamless import operations without affecting the UI responsiveness.

Key Features

Custom Management Command: Utilizes Django's custom command to initiate and manage data imports from various sources.
MongoDB Integration: Incorporates MongoDB as the database backend for storing imported data.
Asynchronous Processing: Uses Celery and Redis to handle import tasks asynchronously, allowing the UI to remain responsive during large file imports.
Progress Tracking: Provides real-time progress updates and status notifications for ongoing import operations.
Error Handling: Implements robust error handling mechanisms to manage exceptions during the import process.
Logging: Utilizes logging to maintain detailed records of import activities and errors.

Installation

Clone the repository:

https://github.com/Muneeb1030/Django_DataImporter.git
cd Django_DataImporter

Install dependencies:

pip install -r requirements.txt

Configure MongoDB:

Ensure MongoDB is installed and running.
Update database settings in settings.py to connect to your MongoDB instance.

Configure Redis:

Ensure Redis is installed and running.
Update Celery settings in settings.py to configure Redis as the broker and backend.

Apply database migrations:

python manage.py migrate

Start Celery Worker:

celery -A your_project worker -l info

Run Django development server:

python manage.py runserver

Access the application at http://localhost:8000/.

Usage

Data Import: Execute the custom command to initiate data imports:

python manage.py import_data --file path_to_your_data_file.csv