Skip to content

feat(devices): implement bulk device import with column mapping, live preview, and inline banners (vertex app)#3544

Merged
Baalmart merged 18 commits into
stagingfrom
ft-bulk-import
May 27, 2026
Merged

feat(devices): implement bulk device import with column mapping, live preview, and inline banners (vertex app)#3544
Baalmart merged 18 commits into
stagingfrom
ft-bulk-import

Conversation

@Codebmk

@Codebmk Codebmk commented May 27, 2026

Copy link
Copy Markdown
Member

Overview

This PR introduces a comprehensive bulk device import feature allowing administrators and operators to upload multiple devices at once via CSV or JSON files. It extends the ImportDeviceModal to provide an interactive, multi-step flow that guides users through file parsing, intelligent column mapping, data preview, and batch processing with downloadable error logs for failed rows.

To align with premium UI/UX guidelines, all notifications related to bulk import are routed through the context-aware useBanner system, displacing raw floating toasts in favor of scoped inline banner alerts directly inside the modal container.

Key Features

  1. Multi-Format Support & File Upload: Added a new reusable file uploader (ReusableFileUpload) accepting .csv (parsed client-side using papaparse) and .json files.
  2. Column Field Mapping: Automatically maps uploaded file columns to expected fields (long_name, serial_number, latitude, longitude, api_code, description, and device_number) using common aliases. Provides dropdowns for manual mapping overrides.
  3. Global Config Overrides: Allows users to specify a sensor manufacturer (network), category, and tags that apply to all imported devices in the batch.
  4. Data Validation & Live Preview: Displays a tabular preview of the first 5 transformed devices for validation before committing the import.
  5. Inline Scoped Banners & Result Logs: Uses the centralized useBanner context to display success, warning, or error alerts inline inside the modal header. Provides a summary breakdown table and supports downloading a custom failed_devices.csv containing only failed rows and error reasons.
  6. Robust Mutator Hook: Built the clean useBulkImportDevices mutation hook to interface with the /devices/soft/bulk endpoint, handle HTTP 207 responses, and invalidate cache keys for local and admin device grids.

Changes Implemented

1. Types & Data Structures

  • devices.ts: Added TypeScript interfaces for BulkImportDeviceResult and BulkImportDeviceResponse to structure the backend batch response.

2. API Services & Hooks

  • devices.ts: Added importBulkDevicesCSV and importBulkDevicesJSON methods under devices api client to submit imports through the /devices/soft/bulk API endpoint.
  • useDevices.ts: Created useBulkImportDevices hook that handles mutation execution and invalidates React Query state keys. Removed all ReusableToast calls.

3. UI & Components

  • import-device-modal.tsx: Overhauled the import dialog to manage multi-step bulk upload state machine (Upload -> Map Fields -> Preview -> Results/Errors) and display inline scoped banners on mutation resolution.
  • ReusableFileUpload.tsx [NEW]: Created a reusable drag-and-drop / click-to-select file upload input widget.
  • ReusableInputField.tsx & ReusableSelectInput.tsx: Standardized style of required asterisks (*) to use text-red-500 for consistent design alignment.

4. Dependencies

  • package.json: Updated @types/papaparse version to ^5.5.2 for type safety in CSV parsing.

5. Documentation

  • changelog.md: Updated with the Version 1.23.50 entry detailing the new feature and notification system changes.

Testing & Verification

  • Single Upload & Basic Flow: Verified CSV parse maps file headers to form fields.
  • Preview & Mapping Step: Verified manual column selection updates mapped preview data.
  • Bulk Creation Mutation: Verified call to backend bulk import endpoint with JSON payload.
  • Inline Banner Alerts: Verified that complete success triggers a global success banner and closes the modal, whereas partial success or failure renders the correct inline alert with a detailed summary inside the modal.
  • Partial Failures & Error Handling: Simulated partial imports and verified the "Download Failed CSV" correctly constructs and downloads failed_devices.csv.
  • Grid Cache Invalidation: Verified that on successful import, device lists automatically refresh.

Screenshots (optional)

Screenshot 2026-05-27 133627 Screenshot 2026-05-27 133643 Screenshot 2026-05-27 133722 Screenshot 2026-05-27 133731 Screenshot 2026-05-27 133749 Screenshot 2026-05-27 133616

Summary by CodeRabbit

  • New Features
    • Device bulk import feature with CSV/JSON file upload support
    • Interactive column auto-mapping with override capability
    • Preview of uploaded data before import
    • Validation with inline error reporting
    • Downloadable report of failed imports
    • Shared settings can be applied across imported device batch
    • Enhanced required field indicators in form inputs

Review Change Stack

Codebmk and others added 12 commits May 13, 2026 22:16
Replace ReusableToast notifications with context-aware banners for bulk device imports. The bulk import hook (useBulkImportDevices) no longer surfaces toasts or handle error/success UI; it delegates notification responsibility to callers and only invalidates relevant queries. ImportDeviceModal now uses useBanner (showBanner/hideBanner) to display inline success, warning, and error alerts, handles partial/failure cases, offers downloadable failed_devices.csv, and hides banners when the modal closes. Updated changelog to reflect the new banner-based feedback behavior.
@Codebmk Codebmk self-assigned this May 27, 2026
@coderabbitai

coderabbitai Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

@Codebmk, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 40 minutes and 36 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 730bd599-9dba-40df-837e-9890c16430fa

📥 Commits

Reviewing files that changed from the base of the PR and between 9aa8d08 and 016428d.

📒 Files selected for processing (5)
  • src/vertex/components/features/devices/import-device-modal.tsx
  • src/vertex/components/shared/fileupload/ReusableFileUpload.tsx
  • src/vertex/components/shared/inputfield/ReusableInputField.tsx
  • src/vertex/components/shared/select/ReusableSelectInput.tsx
  • src/vertex/core/apis/devices.ts
📝 Walkthrough

Walkthrough

This PR implements a complete device bulk import feature. Users can now upload CSV or JSON files, auto-map columns to device fields, preview the parsed data, and submit bulk imports with optional network overrides and batch-wide settings. The implementation includes new type contracts, a file upload component, API client methods, a React Query hook, modal state management, and standardized required-field styling.

Changes

Device Bulk Upload

Layer / File(s) Summary
Bulk import type contracts
src/vertex/app/types/devices.ts
BulkImportDeviceResult and BulkImportDeviceResponse interfaces define per-device outcomes and overall import summary including success/failure counts and detailed per-device results.
File upload component
src/vertex/components/shared/fileupload/ReusableFileUpload.tsx
New ReusableFileUpload component provides accessible file input with optional label, description, error messaging, and onChange callback forwarding.
API client and bulk import hook
src/vertex/core/apis/devices.ts, src/vertex/core/hooks/useDevices.ts
devices.importBulkDevicesCSV() and devices.importBulkDevicesJSON() API methods post multipart/JSON to /devices/soft/bulk; useBulkImportDevices hook wraps the mutation and invalidates device/network query caches on success.
Bulk import modal implementation
src/vertex/components/features/devices/import-device-modal.tsx
ImportDeviceModal now supports file upload via PapaParse, auto-maps expected device fields (with case-insensitive matching and aliases), transforms parsed rows for preview, builds bulk payloads with user/cohort/network override, displays results, and generates downloadable CSV of failed rows.
Required field indicator styling
src/vertex/components/shared/inputfield/ReusableInputField.tsx, src/vertex/components/shared/select/ReusableSelectInput.tsx
Standardize required-field asterisk rendering to use text-red-500 instead of theme-variable-based colors.
Changelog and dependencies
src/vertex/app/changelog.md, src/vertex/package.json
Document bulk upload feature, API/hook additions, UI updates, and file changes; bump @types/papaparse to ^5.5.2.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • airqo-platform/AirQo-frontend#3503: Modifies ImportDeviceModal submission and error/feedback flows; may have code-level conflicts or require coordination with bulk import changes.

Suggested labels

ready for review

Suggested reviewers

  • Baalmart
  • OchiengPaul442

Poem

📦 Files now flow in bulk through fields so neat,
Mapped columns dance, with CSV beat,
Preview the rows before they fly,
Failed ones download—no need to cry! 🚀

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main feature: bulk device import with column mapping, live preview, and inline banners for the vertex app.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ft-bulk-import

Warning

Review ran into problems

🔥 Problems

Errors were encountered while retrieving linked issues.

Errors (1)
  • JIRA integration encountered authorization issues. Please disconnect and reconnect the integration in the CodeRabbit UI.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions

Copy link
Copy Markdown
Contributor

New azure vertex changes available for preview here

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/vertex/components/features/devices/import-device-modal.tsx`:
- Around line 230-252: The preview/import allows the same source column to be
mapped to multiple expected fields; update the validation before building
transformedDevices to detect duplicate mappings in fieldMapping (e.g., collect
Object.values(fieldMapping) excluding falsy, find duplicates) and if any
duplicates exist call setErrors with a clear message listing the duplicated
source headers (and optionally which expected fields collide), then return
early; place this check after computing missingRequired and before using
EXPECTED_FIELDS/parsedData to build transformedDevices so duplicates cannot
produce silent incorrect imports.
- Around line 142-160: The autoMapFields function currently only
lowercases/trims headers, so variants like "Device Name", "device-name", or
"serial number" are missed; update autoMapFields to normalize headers and keys
before matching by converting to lowercase, trimming, replacing non-alphanumeric
characters (spaces, hyphens, punctuation) with underscores, collapsing multiple
underscores, and trimming leading/trailing underscores, then build a
normalizedHeaders map from normalized -> original header; normalize each
EXPECTED_FIELDS key (and apply the special aliases for long_name and
serial_number using normalized forms) and use normalizedHeaders to find matches
and populate initialMapping[field.key] with the original header name; modify the
lowerHeaders variable and matching branches in autoMapFields (and the
long_name/serial_number alias logic) to use the normalized forms so common
export variants auto-map correctly.
- Around line 181-191: When parsing CSVs in import-device-modal.tsx (the
Papa.parse callback used when fileName.endsWith('.csv')), check that
results.data contains at least one non-empty row before enabling mappingMode:
call setFileHeaders(headers) and setParsedData(results.data) as you do, but only
call autoMapFields(headers) and setMappingMode(true) if results.data.length > 0
(or after filtering out empty rows). If there are zero device rows, set a
file-level validation error state (do not call autoMapFields or setMappingMode)
so the UI stays in file-selection/error state and the Preview Import path does
not fall through to validateForm().

In `@src/vertex/components/shared/fileupload/ReusableFileUpload.tsx`:
- Around line 53-62: The hidden file input in ReusableFileUpload.tsx (the <input
id={inputId} type="file" ... />) must be cleared after each selection so
choosing the same file again still triggers onChange; update the onChange
handler used on that input (the inline arrow handling e => { if (onChange)
onChange(e.target.files?.[0] || null); }) to also reset the native input value
(e.currentTarget.value = '' or e.target.value = '') after invoking onChange (or
nulling it before if you prefer) so the browser will fire change events for the
same file on subsequent attempts.

In `@src/vertex/components/shared/select/ReusableSelectInput.tsx`:
- Line 189: The required-field asterisk in ReusableSelectInput is using a
hardcoded "text-red-500"; update the span rendering inside the
ReusableSelectInput component to use the shared token/class used elsewhere
(e.g., "text-destructive") so the required indicator matches other select/input
components; replace the class on the {required && <span ...>*</span>} element
accordingly to the agreed shared token.

In `@src/vertex/core/apis/devices.ts`:
- Around line 463-470: The headers object passed to jwtApiClient.post for the
`/devices/soft/bulk` request is hardcoding 'Content-Type':
'multipart/form-data', which prevents Axios from adding the required boundary;
update the call in devices.ts (the jwtApiClient.post invocation) to remove the
'Content-Type' header while keeping 'X-Auth-Type': 'JWT' so Axios can infer and
set the correct multipart boundary from the FormData payload.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9bb87309-82a8-4937-95d1-97a92381fb2c

📥 Commits

Reviewing files that changed from the base of the PR and between ed03ef7 and 9aa8d08.

⛔ Files ignored due to path filters (1)
  • src/vertex/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (9)
  • src/vertex/app/changelog.md
  • src/vertex/app/types/devices.ts
  • src/vertex/components/features/devices/import-device-modal.tsx
  • src/vertex/components/shared/fileupload/ReusableFileUpload.tsx
  • src/vertex/components/shared/inputfield/ReusableInputField.tsx
  • src/vertex/components/shared/select/ReusableSelectInput.tsx
  • src/vertex/core/apis/devices.ts
  • src/vertex/core/hooks/useDevices.ts
  • src/vertex/package.json

Comment thread src/vertex/components/features/devices/import-device-modal.tsx
Comment thread src/vertex/components/features/devices/import-device-modal.tsx
Comment thread src/vertex/components/features/devices/import-device-modal.tsx
Comment thread src/vertex/components/shared/fileupload/ReusableFileUpload.tsx
Comment thread src/vertex/components/shared/select/ReusableSelectInput.tsx Outdated
Comment thread src/vertex/core/apis/devices.ts Outdated
Codebmk added 6 commits May 27, 2026 14:06
Enhance autoMapFields in ImportDeviceModal to better match CSV headers by normalizing values (toLowerCase, trim, and remove non-alphanumeric chars) and using a normalizedHeaders array. Add alias handling for long_name and serial_number (e.g. locationname, devicename, serial, id) and simplify lookup logic for more robust automatic field mapping.
Add a guard in ImportDeviceModal to detect when the parsed CSV contains no data rows. Extract parsed rows into a variable, and if rows.length === 0 clear headers/parsedData/fieldMapping, disable mapping mode, set a general error ('The uploaded CSV does not contain any devices.'), and return early. Also use the rows variable when setting parsed data.
Add a check in ImportDeviceModal to detect and prevent duplicate mappings of file columns to device fields. The code collects mapped headers from fieldMapping, finds duplicates, and sets a general error "Each file column can only be mapped once." and aborts the import flow if duplicates are found. This prevents incorrect/ambiguous imports caused by mapping the same CSV column multiple times.
Refactor the file input change handler: compute nextFile using nullish coalescing and invoke the callback with optional chaining (onChange?.(nextFile)). Also clear e.currentTarget.value after handling so the same file can be selected again without issues. This is a small behavioral change to allow re-uploading the same file and a code cleanup.
Replace hardcoded Tailwind color class `text-red-500` with the semantic `text-destructive` utility for required field asterisks. Changes applied to ReusableInputField and ReusableSelectInput to ensure consistent styling and better theme/design token support.
Removed the explicit 'Content-Type: multipart/form-data' header from the JWT API client's POST request for soft bulk device import. This allows the HTTP client to set the multipart boundary automatically and prevents incorrect Content-Type issues; the 'X-Auth-Type: JWT' header is retained.
@github-actions

Copy link
Copy Markdown
Contributor

New azure vertex changes available for preview here

@Baalmart
Baalmart merged commit d4b5ea4 into staging May 27, 2026
21 checks passed
@Baalmart
Baalmart deleted the ft-bulk-import branch May 27, 2026 12:19
@Baalmart Baalmart mentioned this pull request May 27, 2026
7 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants