Handling Large CSV Files During Import

AccuArk can handle CSV files with thousands of rows, but large imports require a bit more planning than small ones. This article explains how AccuArk processes large files, what to expect during the import, and how to avoid common problems with big datasets.

How AccuArk Processes Large Files

AccuArk processes imports in batches of 100 rows. Each batch is committed to the database individually, which means:

• If a batch succeeds, those rows are saved permanently
• If a batch fails, only that batch is rolled back — previously committed batches are not affected
• The progress bar and import log update in real time so you can track how far along the import has progressed

This batch approach protects your data and allows you to recover easily from errors without losing all of your progress.

What to Expect During a Large Import

• Do not close the wizard window while the import is running. Closing the window will cancel the remaining batches. Rows that were already committed will remain in the database, but uncommitted rows will be lost.
• The progress bar shows real-time status. You can see how many rows have been processed, how many succeeded, and how many failed as the import runs.
• Large imports take time. A file with 5,000 or more rows may take several minutes to complete, depending on the data type. Data types that involve multiple lookup validations (such as inventory items or invoices) take longer per row than simpler types (such as categories or chart of accounts).
• Your computer may feel slower during very large imports. The import process uses system memory and network bandwidth to communicate with the database. Avoid running other resource-intensive applications at the same time.

Handling Errors in Large Files

If errors occur during a large import, you have two strategies for handling them:

• Fix and re-import the full file — For data types that support upsert (such as Inventory Items, Customers, and Vendors), you can safely re-import the corrected file. Rows that were already imported successfully will be matched by their key field and updated, so you will not create duplicates.
• Import only the failed rows — After the import finishes, click Export Error Report to get a CSV containing only the rows that failed. Fix the errors in that file and import it as a separate file. This is faster than re-importing the entire dataset.

If the data type does not support upsert, use the error report approach to avoid creating duplicate records.

Splitting Large Files

If you prefer to work with smaller files, you can split your CSV into chunks of 1,000 to 2,000 rows each and import them one at a time. This approach offers several advantages:

• Easier to identify and fix errors in a smaller dataset
• Each import completes faster, so you get feedback sooner
• Less memory usage during parsing and validation

When splitting, make sure each file includes the header row as the first row. AccuArk uses the header row to identify columns during the mapping step.

Excel Pitfalls with CSV Files

Microsoft Excel can inadvertently alter your data when saving to CSV. Be aware of these common issues:

• Leading zeros are removed — Excel treats numbers as numeric values by default, which strips leading zeros from ZIP codes, phone numbers, and account codes. For example, the ZIP code “07102” becomes “7102.” To prevent this, format those columns as Text in Excel before entering or pasting data.
• Long numbers are converted to scientific notation — Numbers longer than 15 digits (such as certain barcodes or serial numbers) may be displayed as scientific notation (e.g., “1.23457E+15”). Format the column as Text to preserve the full number.
• Dates may be reformatted — Excel may change date formats based on your system locale. If your import requires a specific date format, verify the dates in the CSV after saving.

To avoid all of these issues, consider formatting the entire worksheet as Text before pasting your data, or use a text editor to verify the final CSV content.

File Encoding

For best compatibility with special characters (accented letters, currency symbols, and non-English text), save your CSV file with UTF-8 encoding. In Excel, select CSV UTF-8 (Comma delimited) (*.csv) as the file type when saving. In Google Sheets, downloaded CSVs are UTF-8 by default. If you see garbled characters after import, re-save the file with UTF-8 encoding and import again.

Very Large Files (50,000+ Rows)

AccuArk loads the entire CSV file into memory during parsing and validation. For extremely large files (50,000 rows or more), this can consume a significant amount of system memory. If you experience slow performance or out-of-memory errors, split the file into smaller chunks before importing.

Tips for Smooth Large Imports

• Validate with a small test batch first — Before importing your full dataset, import just the first 20 to 50 rows to verify that your column mapping is correct and the data validates cleanly.
• Import during off-peak hours — If your business uses AccuArk actively throughout the day, schedule large imports for early morning or after business hours to minimize the impact on other users.
• Close unnecessary programs — Free up system resources by closing other applications, especially those that use significant memory or network bandwidth.
• Keep a backup of your original file — Before making any changes to your CSV, save a copy of the original. This gives you a fallback if something goes wrong during preparation.