Data Migration (ONB04US01)
User Story: AI-Powered Data Migration & Transformation
1. Problem Statement
Utilities migrating from legacy systems to SMART360 face significant challenges in data migration and transformation. The process is:
- Time-consuming, taking weeks to migrate critical utility data such as consumer records, meter readings, and billing history.
- Error-prone, leading to data loss, incorrect meter-to-consumer mapping, and inconsistencies in billing and service requests.
- Manually intensive, requiring extensive effort to clean, format, and validate consumption data, payment records, and asset details.
These challenges directly impact key utility operations, including:
- Billing & Revenue Management – Incorrect data leads to overbilling, underbilling, or missing payments, causing financial discrepancies.
- Service Order Execution – Delays in consumer data migration result in service interruptions and slow request processing.
- Meter Data Management – Inaccurate meter readings and historical data errors affect consumption tracking and billing accuracy.
- Asset & Territory Management – Improper mapping of utility assets (transformers, substations, meters) leads to inefficiencies in maintenance and tracking.
Without an efficient migration system, utilities experience:
- Delayed operational readiness, increasing the time required to onboard and serve customers.
- High support dependency, as utility teams spend excessive time manually fixing migration errors.
- Revenue leakage, where incorrect billing data and missing transactions result in financial losses.
An optimized data migration solution for utilities is essential to ensure accurate, efficient, and seamless onboarding into SMART360.
2. Who Are the Users Facing the Problem?
- Tenant Admin: Unable to migrate data for specific utility
- Utility Admin: Unable to migrate territory data efficiently, leading to delays in system setup.
- CSO Admin: Struggles with bulk uploads of consumer accounts, payments, complaints, and services, causing payment delays and incorrect account statuses.
- MX Manager: Cannot migrate bulk meter data, historical readings, or update large datasets, leading to errors in consumption tracking and billing.
- WX Admin: Lacks bulk import functionality for service order templates, slowing work order execution.
- AX Admin: Faces inefficiencies in importing plant, unit, and asset data, impacting asset tracking and maintenance scheduling.
These users spend weeks cleaning, transforming, and loading data manually using Postman, Excel, and external tools, creating operational bottlenecks.
User Role Permissions for Data Migration
1. Tenant Admin
- Can upload and migrate all tye of data across all utilities.
- Can manage and roll back migrations for all utilities.
2. Utility Admin
- Can upload and migrate data only within their assigned utility.
- Can manage and roll back migrations within their utility within its associated utility.
3. CSO Admin (Consumer Data Management)
- Can upload and migrate consumer related data (accounts, payments, complaints, services) within its associated utility.
- Can manage and roll back consumer data migrations within its associated utility.
4. MX Manager (Meter Data Management)
- Can upload and migrate meter related data (meters, historical readings) within its associated utility.
- Can manage and roll back for meter data migrations within its associated utility.
5. BX Manager (Billing Data Management)
- Can upload and migrate Billing related data within its associated utility.
- Can manage and roll back for Billing data migrations within its associated utility.
6. O&M Manager (Work & Service Orders)
- Can upload and migrate service orders related data within its associated utility.
- Can manage and roll back service order migrations within its associated utility.
7. AX Admin (Assets & Plant Data)
- Can upload and migrate assets related data within its associated utility.
- Can manage and roll back asset data migrations within its associated utility.
3. Jobs To Be Done
As a Utility Admin / CSO Admin / MX Manager / BX Manager / O&M Manager / AX Admin, I am responsible for managing vast amounts of consumer, meter, and billing data. Migrating this data from legacy systems to SMART360 has always been a challenge—manual processes are slow, error-prone, and disrupt daily operations.
I need a seamless and intuitive bulk data migration tool that allows me to upload large datasets effortlessly while ensuring accuracy and compliance. The system should accurately and automatically map fields, validate data, and highlight errors before final import, giving me confidence that the data is clean and reliable.
With this solution, I can focus on operations instead of data corrections, reduce migration time, and ensure a smooth transition to SMART360. This will improve efficiency, drive higher adoption, and allow my team to continue working without disruptions.
4. Solution
The AI-Powered Data Migration & Transformation feature introduces:
- AI-Powered Column Mapping – Auto-detects & suggests field mappings with accuracy scores.
- Data Transformation Rules – Standardizes formats, merges/splits columns, and applies regex corrections.
- Live Validation & Error Handling – Detects missing/incorrect data with inline fixing suggestions.
- Smart Auto-Correction & Recommendations – AI-powered anomaly detection and bulk edit options.
- Real-Time Progress & Rollback – Track migration progress, pause, retry, or rollback if needed.
- Post-Migration Reports & Audit Logs – Exportable logs, success/failure reports, and full audit trails.
5. Process Flow
6. Major Steps Involved
- User uploads bulk data file (CSV).
- AI-powered column mapping suggests auto-matched fields with manual override.
- Data transformation engine cleans and formats entries (e.g., date standardization, merging duplicate records).
- Live validation detects issues (missing fields, incorrect formats, duplicate entries).
- Inline fixing suggestions are provided for errors.
- Preview step allows users to compare source vs. transformed data.
- User initiates migration, tracking real-time progress.
- Audit logs & post-migration reports confirm success/failures and rollback options.
7. Business Rules
Process Authorization Rules
Upload Process
- The Utility should be selected for the process to begin.
- The system supports AI-powered data upload functionality for seamless integration into the Smart360 system.
- Users can upload data for various data types including:
- Consumers (customer accounts and profiles)
- Meters (meter information and details)
- Meter Data (usage readings and consumption data)
- Plans (service plans and rate configurations)
- Tariffs (pricing structures and rate cards)
- Service Orders (work orders and service requests)
- Payments (customer payment history and transactions)
- Complaints (customer support tickets and issues)
- Plant (Plant configuration data)
- Unit (Unit configuration data)
- Asset(Asset configuration data)
- User Data (system users and administrative accounts)
- File upload supports CSV formats
- The system should allow a maximum of 50,000 rows per file upload to ensure performance optimization and prevent system overload.
- The system provides a preview of the uploaded data showing a sample of records (e.g., "Showing 5 of 7052 records").
- Files can be uploaded either through drag-and-drop functionality or by browsing for files.
- User can download the data template file in CSV format
- The template will contain:
- Predefined column headers based on the selected data type.
- Sample data in the first few rows to help users understand the expected format.
- The system displays upload progress with percentage completion indicators.
- Once uploaded, the system displays a confirmation message for successful uploads.
Data Analysis and Mapping
- AI automatically analyzes the file and suggests column mappings between source columns and target Smart360 fields.
- Each suggested mapping is assigned a confidence level (High, Medium, Low) to indicate the reliability of the mapping.
- File information is displayed showing:
- Filename
- Total columns (e.g., 8)
- Total rows (e.g., 2,426)
- Mapping quality indicators (e.g., 5 High, 2 Medium, 1 Low)
- The system evaluates the uploaded file and assigns confidence levels based on three key factors:
Confidence Level | Criteria | Example |
---|---|---|
High ✅ | - Exact Name Match between the uploaded field and system field. - Exact Data Type Match (e.g., String → String, Date → Date, Integer → Integer). - Strong Pattern Similarity (e.g., Phone numbers, ZIP codes). | Uploaded File Field:
✅ High confidence because the name matches exactly and the expected data type is correct. |
Medium ⚠️ | - Partial Name Match (e.g., synonyms, abbreviations, slight variations). - Compatible but Different Data Types (e.g., Integer vs. String for an ID field). - Pattern Recognition with Minor Variations (e.g., "Meter No." vs. "Meter ID"). | Uploaded File Field:
⚠️ Medium confidence because "Cust_ID" is a partial match for "Consumer ID", but it's not exact. |
Low ❌ | - No Name Similarity between the uploaded field and system field. - Incompatible Data Types (e.g., Text mapped to Date). - Unrecognized Field Content (e.g., a completely unrelated field). | Uploaded File Field:
❌ Low confidence because "Billing Address" is unrelated to "Consumer ID". |
- Example Scenario
A user uploads a file containing the following columns:
Uploaded Field | System Field (SMART360) | Confidence Level | Reason |
---|---|---|---|
|
| High ✅ | Exact name and data type match. |
|
| Medium ⚠️ | Partial name match, slightly different format. |
|
| Medium ⚠️ | Recognized abbreviation, slightly different spelling. |
|
| Low ❌ | No strong similarity between names. |
|
| Low ❌ | Incorrect field mapping; different meanings. |
- Source columns are mapped to target fields with sample data displayed for reference.
- Users can manually adjust mappings through dropdown selection if the AI suggestion is incorrect.
- The system enforces proper field type mapping (e.g., ensuring Customer ID fields map to appropriate ID fields).
Validation Process
- The system performs data validation after mapping to identify issues requiring attention.
- Validation results are categorized as:
- Valid records (with percentage)
- Warnings (with percentage)
- Errors (with percentage)
- The system specifically identifies and flags:
- Format issues (e.g., "Missing Postal Code format")
- Data inconsistencies (e.g., "Date format inconsistency")
- Unmapped fields (e.g., "Unmapped field: Meter_Type")
- For each issue type, the system shows the number of affected records.
- Users can download a validation report for offline review.
- The system provides detailed information for each issue, including specific records affected.
- A "Fix All Issues" option is available for batch resolution of problems.
Handling Missing Primary Keys for Data Upload
- If the uploaded file does not contain the required primary key for the selected dataset, the system will take action based on the Onboarding (ONB) settings.
Handling Scenarios
1. ONB Has Predefined Prefix & Starting Number
- If the prefix and starting number are already set in ONB, the system will:
- Auto-fill the missing primary key using the predefined prefix and starting number.
- Generate unique primary keys in sequence.
- Show a preview for confirmation before migration.
Example:
- Dataset: Consumer Data
- ONB Settings: Prefix =
CUST
, Starting Number =1001
- Generated Keys:
CUST1001
,CUST1002
,CUST1003
… - User Options:
- Confirm the auto-filled primary keys.
- Re-upload the file if the user wants to provide their own primary keys.
2. ONB Does Not Have Predefined Prefix & Starting Number
- If ONB settings do not have a predefined prefix and starting number:
- The system will prompt the user to choose between:
- Manual Setup: Enter a prefix and starting number.
- AI-Assisted Setup: Auto-generate primary keys based on sequential logic.
- The system will display a preview before finalization.
- The system will prompt the user to choose between:
Primary Keys Required by Dataset
Dataset Type | Primary Key Required |
---|---|
Consumer Data | Customer Number |
Meter Data | Meter Number |
Asset Data | Asset ID, Plant ID, Unit ID |
- Auto-fill when ONB settings exist to ensure seamless migration.
- Prompt user only when ONB settings are missing, reducing unnecessary steps.
- Uniqueness Check: Ensure no duplicate keys are generated.
- Preview & Confirmation: Users must verify generated primary keys before finalizing.
- Re-upload Option: If users want to provide their own primary keys, they can re-upload the file.
Data Correction
- Users can view detailed information about each issue by expanding the issue sections.
- For format issues (e.g., postal codes), the system allows:
- Viewing specific invalid records
- Auto-formatting data to meet system requirements
- For unmapped fields, the system:
- Suggests possible Smart360 mappings
- Allows users to select the appropriate field mapping
- The system tracks correction status, showing:
- Fixed issues (with green checkmarks)
- Remaining issues (with warning or error icons)
- When issues are corrected, the system updates validation statistics (e.g., Valid records percentage increases).
- Auto-format functionality can standardize data formats (e.g., converting postal codes to standard formats).
Error Handling
- Unexpected errors must be flagged for immediate administrator attention
- The system must implement automatic retry logic for transient errors
- Critical errors must halt the migration process until administrator intervention
Error messages must be specific and provide examples for corrections.
✅ Example of Effective Error Handling:
Error Type | Message | Suggested Fix |
---|---|---|
Incorrect Format | Date format inconsistent: "Jan 5 2021", "02/03/2024". | Use the standard format: YYYY-MM-DD (e.g., 2024-03-20) . |
Missing Field | Meter ID missing in row 5. Required for meter mapping. | Enter a valid Meter ID before proceeding. |
Duplicate Entry | Meter ID "5001" already exists in the uploaded data (Row 6, Row 12). | Remove duplicates from the uploaded file before re-uploading. |
Overwriting Data | Existing consumer data for Consumer ID "C123" found. | Confirm if data should be updated or kept unchanged. |
Invalid Value | Negative consumption (-250 kWh) in row 12. | Verify meter readings and enter a positive value. |
Mapping Error | Meter ID "M1002" not linked to any consumer account. | Ensure meters are mapped to a valid consumer. |
Unsupported Character | Special characters found in Consumer Name ("John@Doe"). | Remove special characters; use only letters and spaces. |
AI Assistant Integration Rules
- AI assistant must provide real-time updates on migration progress
- Assistant must offer troubleshooting guidance for common error patterns
- AI recommendations must be reviewed by human operators before implementation
- Assistant should suggest optimization strategies based on observed performance
Migration Process
- The system tracks total records to be processed (2897 in total)
- Current progress is displayed both numerically (375 of 2897) and as a percentage (13% Complete)
- The progress bar visually represents completion percentage
- The system maintains counts of successful migrations (367) and errors (8)
- Errors are logged in a separate "Error Log" section with row numbers and error messages
- All migration activity is timestamped [HH:MM PM]
- Log entries show both successful and failed operations
- The system captures the specific row numbers where errors occur
- All errors currently show as "Unexpected error during migration" suggesting generic error handling
Migration Reporting Rules
- Each migration must generate a timestamped report (format: MM/DD/YYYY, HH:MM
AM/PM) - Reports must include total record count, success rate, error rate, and processing time
1. Total Record Count
Definition: The total number of rows processed from the uploaded file.
Formula: Total Record Count=Number of Rows in the Uploaded File\text{Total Record Count} = \text{Number of Rows in the Uploaded File}Total Record Count=Number of Rows in the Uploaded File
Example: If a user uploads a file with 50,000 rows, the Total Record Count = 50,000.
2. Success Rate (%)
Definition: The percentage of records successfully migrated without errors.
Formula: Success Rate=(Successfully Migrated RecordsTotal Record Count)×100\text{Success Rate} = \left( \frac{\text{Successfully Migrated Records}}{\text{Total Record Count}} \right) \times 100Success Rate=(Total Record CountSuccessfully Migrated Records)×100
Example:
Uploaded file: 50,000 rows
Successfully migrated records: 47,500 rows
Success Rate = (47,500 / 50,000) × 100 = 95%
3. Error Rate (%)
Definition: The percentage of records that failed due to validation errors, incorrect field mappings, or missing data.
Formula: Error Rate=(Failed RecordsTotal Record Count)×100\text{Error Rate} = \left( \frac{\text{Failed Records}}{\text{Total Record Count}} \right) \times 100Error Rate=(Total Record CountFailed Records)×100
Example:
Uploaded file: 50,000 rows
Failed records due to errors: 2,500 rows
Error Rate = (2,500 / 50,000) × 100 = 5%
4. Processing Time
Definition: The total time taken to process and migrate the data.
Formula: Processing Time=Migration End Time−Migration Start Time\text{Processing Time} = \text{Migration End Time} - \text{Migration Start Time}Processing Time=Migration End Time−Migration Start Time
Example:
Migration started at 10:00 AM
Migration completed at 10:05 AM
Processing Time = 5 minutes - Success/error distribution must be visually represented with color-coded indicators
- Users must be able to return to the dashboard from the report view
Error Report Generation
- The Error Report is generated when data migration fails for certain records.
- The report is downloadable in Excel (CSV) format.
- Each row in the report corresponds to a record that failed migration.
- The report should clearly indicate why each record was not migrated.
- If no errors are found, the download option is disabled.
- The downloaded CSV must include a detailed message explaining why each data entry failed.
- Show the originnal data entered and then add a error discription coloum.
- The Error report will only include non migrated data with the error message.
Error Report Sample
Consumer No | Meter No | Premise | Error Description | |
---|---|---|---|---|
(Blank) | 123456 | 123 Main Street | Consumer No is required and cannot be empty. | |
98765 | ABC123 | user@xyz | 456 Oak Avenue | Meter No must be numeric, but received 'ABC123'. |
12345 | 789012 | user@invalid | 789 Pine Road | Invalid email format. Expected format: xyz@domain.com. |
54321 | 654321 | 321 Birch Lane | Premise not found or not related to the consumer. | |
11223 | 987654 | 852 Cedar Road | Negative balance is not allowed for active accounts. |
Migration History Rules
- The system must maintain a log of all migrations with date and type information
- Migrations must be searchable by name or other parameters
- Migrations must be filterable by:
- Data format type (CSV)
- Data category, eg: consumer data
- New migrations must be clearly identified in the history
- Migration history must display the most recent migrations at the top
- Migration names must be descriptive of their content when specified by users
- On view for completed migrations, it will redirect to the migration report.
Security and Data Management
- The system tracks and allows viewing of upload history.
- The system appears to enforce data quality standards through validation rules.
- The system maintains the relationship between related data entities (e.g., Customer IDs and Meter IDs).
- The system enforces format standards for critical fields like postal codes, dates, and IDs.
- The system allows for error handling and recovery from validation failures.
8. Sample Data Format
9. Acceptance Criteria
- Users can successfully upload bulk data files with 100% accuracy.
- AI-powered field mapping works with accuracy.
- Live validation must detect incorrect/missing data.
- Users can manually correct errors before migration.
- Preview functionality allows comparison between original & processed data.
- Migration progress can be tracked with a real-time status.
- Audit logs provide detailed insights on data corrections.
10. Process Changes
- Eliminates Postman/manual scripts for data import.
- Integrates AI-driven validation before actual migration.
- Replaces manual transformation steps with automated rule-based corrections.
- Reduces dependency on customer support for migration issues.
11. Impact from Solving This Problem
Impact Metrics: SMART360 Data Migration Module
Metric | Before Implementation (Legacy System Challenges) | After Implementation (SMART360 Solution) |
---|---|---|
Time to Migrate a Single Data Type | 2-4 weeks per data type due to manual validation and cleaning. | Reduced to 1-2 hours with automated validation and transformation. |
Data Errors (Formatting, Duplicates, Missing Fields) | 20-30% of uploaded records required manual fixes. | <5% error rate with structured validation and predefined templates. |
Operational Readiness Delays | Onboarding delayed by 4-6 weeks due to incomplete or incorrect data. | Onboarding within 1 weeks , ensuring a faster go-live. |
Support Dependency for Data Fixes | 100% of migration cases required customer support intervention. | <20% support dependency , as most errors are resolved automatically. |
Billing Inaccuracy Due to Data Issues | Up to 25% of bills generated with incorrect consumption or missing data. | <1% billing errors , ensuring accurate invoicing. |
12. Summary of Business Impact
By implementing the SMART360 Data Migration Module, utilities can expect:
✅ 80% faster data migration
✅ 85% reduction in errors
✅ 60% lower dependency on manual fixes
✅ Faster system onboarding (within weeks instead of months)
✅ Improved billing accuracy, reducing revenue leakage
13. User Behavior Tracking
1. Tenant Admin
Objective: Track Tenant Admins managing data migration across the utilities
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Data Uploads | How frequently bulk data is uploaded | Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether fields are mapped correctly | Fields Mapped |
,
,
|
Validation Errors | How many records fail validation | Data Validation |
,
,
,
|
Data Review & Approval | How often records are reviewed before migration | Data Approved |
,
,
,
|
Migration Execution | How often data is successfully migrated | Data Migrated |
,
,
,
|
Key Insights:
- Are Tenant Admins uploading data efficiently?
- How frequently do mapping errors occur?
- How long does it take to validate and approve data before migration?
2. Utility Admin
Objective: Track Utility Admins managing data migration for their assigned utility.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Data Uploads | Frequency of data uploads within a utility | Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether fields are mapped correctly | Fields Mapped |
,
,
|
Validation Errors | How many records fail validation | Data Validation |
,
,
,
|
Data Review & Approval | How often records are reviewed before migration | Data Approved |
,
,
,
|
Migration Execution | How often data is successfully migrated | Data Migrated |
,
,
,
|
Key Insights:
- How efficiently are Utility Admins managing data migration?
- Are validation issues slowing down migration?
- How long does the approval process take before migration?
3. CSO Admin (Consumer Data Management)
Objective: Track CSO Admins handling consumer data migration, including accounts, payments, complaints, and services.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Consumer Data Uploads | Frequency of consumer-related data uploads | Consumer Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether consumer data fields are mapped correctly | Consumer Fields Mapped |
,
,
|
Validation Errors | How many consumer records fail validation | Consumer Data Validation |
,
,
|
Data Review & Approval | How often consumer records are reviewed | Consumer Data Approved |
,
,
,
|
Migration Execution | How often consumer data is migrated successfully | Consumer Data Migrated |
,
,
,
|
Key Insights:
- Are consumer records being uploaded and mapped correctly?
- What is the validation failure rate for consumer data?
- How quickly is consumer data approved and migrated?
4. MX Manager (Meter Data Management)
Objective: Track MX Managers overseeing meter data migration, including readings and meter mapping.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Meter Data Uploads | Frequency of meter data uploads | Meter Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether meter fields are mapped correctly | Meter Fields Mapped |
,
,
|
Validation Errors | How many meter records fail validation | Meter Data Validation |
,
,
|
Data Review & Approval | How often meter readings are reviewed and approved | Meter Readings Approved |
,
,
,
|
Migration Execution | How often meter data is migrated successfully | Meter Data Migrated |
,
,
,
|
Key Insights:
- Are meter readings being uploaded and mapped correctly?
- How efficiently are errors being resolved?
- What percentage of data is successfully migrated without validation issues?
5. BX Manager (Billing Data Management)
Objective: Track BX Managers handling billing data migration, including rate structures and billing history.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Billing Data Uploads | Frequency of billing data uploads | Billing Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether billing fields are mapped correctly | Billing Fields Mapped |
,
,
|
Validation Errors | How many billing records fail validation | Billing Data Validation |
,
,
|
Data Review & Approval | How often billing records are reviewed and approved | Billing Data Approved |
,
,
,
|
Migration Execution | How often billing data is migrated successfully | Billing Data Migrated |
,
,
,
|
Key Insights:
- Are billing data uploads happening without major validation errors?
- How efficiently is billing data being reviewed and migrated?
- Are delays in billing approvals affecting migration speed?
6. O&M Manager
Objective: Track O&M Managers handling work order and service order data migration.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Work Order Data Uploads | Frequency of work order data uploads | Work Order Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether work order fields are mapped correctly | Work Order Fields Mapped |
,
,
|
Validation Issues | How many work order records fail validation | Work Order Validation |
,
,
|
Data Review & Approval | How often work orders are reviewed and approved | Work Order Approved |
,
,
,
|
Migration Execution | How often work order data is successfully migrated | Work Order Migrated |
,
,
,
|
Key Insights:
- Are work order records uploaded correctly?
- How efficiently are work order fields being mapped?
- Are approval delays affecting execution?
7. AX Manager
Objective: Track AX Managers managing asset-related data migration, including plant, unit, and asset details.
Metric | What it Tells You | Event to Track | Key Properties |
---|---|---|---|
Asset Data Uploads | Frequency of asset-related data uploads | Asset Data Uploaded |
,
,
,
|
Field Mapping Completion | Whether asset fields are mapped correctly | Asset Fields Mapped |
,
,
|
Validation Errors | How many asset records fail validation | Asset Data Validation |
,
,
|
Data Review & Approval | How often asset records are reviewed and approved | Asset Data Approved |
,
,
,
|
Migration Execution | How often asset data is successfully migrated | Asset Data Migrated |
,
,
,
|
Key Insights:
- Are asset records being uploaded and mapped correctly?
- How frequently do mapping errors occur?
- How long does it take for validation and approval before migration?
14. Wireframe
https://docs.google.com/presentation/d/1MQQYjcWuYSv4zxLWzdrHlI-A2DfqInz3VNRAPcTBt2w/edit?usp=sharing