Designing a Reliable API Integration

1. Define the goal of the integration

Start by describing exactly what the integration should accomplish.

Avoid goals such as:

• Connect the ERP with the e-shop.
• Synchronize the systems.
• Exchange product information.

These descriptions are too broad. They do not explain which data moves, in which direction, or what should happen after it arrives.

A clearer goal would be:

• Synchronize products from the ERP to the e-shop.
• Create products in the e-shop when they are created in the ERP.
• Update prices and stock when they change in the ERP.
• Disable products in the e-shop when they are no longer active in the ERP.

Also define whether the integration is:

• One-way.
• Two-way.
• Real-time.
• Scheduled.
• Manually triggered.
• Event-driven.

Example: The ERP is the source of truth for product codes, names, prices, stock levels, VAT categories, and availability. The e-shop receives these values but does not send product changes back to the ERP.

2. Define the core requirements

Write down the technical and operational requirements before implementing the integration.

Define how frequently the synchronization should run.

Possible options include:

• Immediately after a product changes.
• Every few minutes.
• Every hour.
• Once per day.
• Only when an administrator starts it manually.

The correct frequency depends on the business requirement.

Stock may need to update every few minutes, while product descriptions may only need to update once per day.

Also estimate:

• Number of products.
• Number of changes per day.
• Expected API requests.
• Available processing power.
• API rate limits.
• Maximum acceptable delay.
• Maximum execution time.
• Expected future growth.

Plan the process so it does not overload either system.

Use:

• Pagination when reading data.
• Batches when processing records.
• Controlled concurrency.
• Request timeouts.
• Job timeouts.
• Retry limits.
• Clear loop termination conditions.

Example: The ERP contains 40,000 products, but normally only 300 products change during the day. Instead of processing all 40,000 products every 15 minutes, the integration requests only products modified since the previous successful synchronization.

3. Define and map the fields

List every field that must move between the systems.

For each field, document:

• Source field.
• Destination field.
• Data type.
• Whether it is required.
• Default value.
• Transformation rules.
• Validation rules.

Example mapping:

Do not assume that fields with similar names have the same meaning.

For example, the ERP field quantity may represent physical warehouse stock, while the e-shop field stock_quantity may represent the amount available for online sales after reservations are removed.

Example: If the ERP reports 15 physical units but reserves 3 for existing orders, the integration may need to send 12 available units to the e-shop.

4. Define the source of truth

For every synchronized field, decide which system owns the value.

Without a clear source of truth, the two systems may repeatedly overwrite each other.

For example:

• ERP owns product codes.
• ERP owns prices.
• ERP owns stock.
• ERP owns VAT categories.
• E-shop owns SEO titles.
• E-shop owns product slugs.
• E-shop owns marketing content.

The integration should update only the fields owned by the source system.

Example: The ERP sends the product name, price, and stock. The e-shop administrator writes an SEO title and a longer marketing description. The synchronization must not overwrite those e-shop-specific fields.

For a two-way integration, define ownership for each field separately rather than treating both complete records as mutually editable.

5. Define how records are identified

Every source record must be matched reliably with the correct destination record.

Use a stable unique identifier such as:

• ERP product ID.
• External system ID.
• Immutable SKU.
• UUID.

Do not match products by name. Names can change and may not be unique.

Store the ERP identifier in the e-shop whenever possible.

Example: ERP product 1872 is created in the e-shop and stores 1872 in its erp_product_id field. Future synchronization runs search using this external identifier before creating or updating the product.

If a product code can change, do not use it as the only permanent link between the systems. Use the internal ERP identifier instead.

6. Choose how synchronization starts

The integration may start through:

• Scheduled polling.
• Webhooks.
• A message queue.
• A manual administrator action.
• A combination of these methods.

Polling asks the ERP for changes at regular intervals.

Webhooks allow the ERP to notify the integration when an event occurs. When supported reliably, webhooks can reduce unnecessary polling and provide faster updates.

However, webhook delivery can fail or be repeated. The integration must validate, store, and process webhook events safely.

A practical design may combine methods:

• Use webhooks for quick updates.
• Run a scheduled reconciliation to detect anything the webhooks missed.
• Provide a protected manual synchronization option for administrators.

Example: When a product changes, the ERP sends a product.updated webhook containing the product ID. The integration places the ID in a queue. A background worker retrieves the complete product from the ERP and updates the e-shop. Every night, a scheduled job checks all products changed during the previous day.

7. Design the synchronization workflow

Write down the complete processing sequence before implementing it.

A basic product synchronization workflow may be:

1. Start a synchronization run.
2. Read the previous successful synchronization checkpoint.
3. Request modified products from the ERP.
4. Validate each product.
5. Transform ERP fields into the e-shop format.
6. Find the corresponding e-shop product.
7. Create or update the product.
8. Record the result.
9. Continue to the next batch.
10. Save the new checkpoint after successful processing.
11. Complete the synchronization run.

Separate data retrieval, validation, transformation, and destination updates into clear steps.

This makes failures easier to locate and allows individual parts to be tested independently.

Example: A run starts with checkpoint 2026-07-12 08:00:00. The ERP returns 230 products modified after that time. The integration processes them in three batches and updates the checkpoint only after the required records have been handled successfully.

8. Validate and transform the data

Do not send source data directly to the destination without checking it.

Validate:

• Required fields.
• Data types.
• Allowed values.
• String length.
• Numeric ranges.
• Date formats.
• Currency.
• Tax categories.
• Product status.
• Relationships between records.

Transform values when the two systems use different formats.

Possible transformations include:

• Converting text to the required encoding.
• Trimming whitespace.
• Converting decimal separators.
• Mapping VAT identifiers.
• Converting boolean values.
• Converting date formats.
• Removing unsupported HTML.
• Applying default values.

Invalid data should produce a clear error. It should not silently create an incomplete or incorrect product.

Example: The ERP sends 24_PERCENT as the VAT category, while the e-shop expects the tax class ID standard-24. The integration uses a predefined mapping to convert the ERP value before sending the product.

9. Secure the integration

Run synchronization processes on trusted backend infrastructure.

Do not expose ERP credentials, API keys, client secrets, or synchronization logic directly in browser JavaScript.

Bearer tokens must be protected because anyone possessing a valid token may be able to use its permissions. Tokens and other credentials should be protected both in storage and during transmission.

Store secrets in:

• Environment variables.
• A secret-management service.
• Protected server configuration.

Use:

• HTTPS.
• Short-lived tokens when supported.
• Minimum required API permissions.
• Credential rotation.
• Server-side authorization.
• Protected logs.
• Network restrictions where practical.

If a user interface allows administrators to start synchronization manually, the browser should call a protected backend endpoint. The backend must verify:

• The user is signed in.
• The session or access token is valid.
• The user has permission to start synchronization.
• The request has not been forged.
• Another incompatible synchronization is not already running.

Access control must be enforced in trusted server-side code rather than relying on interface visibility or client-side checks.

Example: The administration page displays a “Synchronize products” button only to authorized users. Pressing it sends an authenticated request to the backend. The backend verifies the administrator permission and places a synchronization job in the queue. The browser never receives the ERP API key.

For incoming webhooks, validate the provider's signature before processing the payload. A shared webhook secret can be used to verify that a request came from the expected provider and was not modified.

10. Control resource usage

Avoid loading or processing the complete dataset in one operation when the data can be divided into batches.

Use:

• API pagination.
• Database pagination.
• Bounded batch sizes.
• Limited parallel processing.
• Memory limits.
• Request timeouts.
• Job timeouts.
• Rate-limit handling.

Batching can reduce the number of operations and improve processing efficiency when the API supports batch actions.

Every loop must have a clear termination condition.

For paginated APIs, verify that:

• The next-page cursor changes.
• Empty pages stop processing.
• The maximum page count is reasonable.
• Repeated cursors are detected.
• Failed pages can be retried safely.

Set timeouts so a failed network request or incorrect loop cannot keep the process running indefinitely. Job-level timeouts can terminate work that becomes stuck or runs much longer than expected.

Example: The integration requests 100 ERP products per page and processes two batches concurrently. Each API request has a 30-second timeout, while the complete synchronization job has a 30-minute limit.

11. Prevent duplicates with idempotency

The same product update may be processed more than once because of:

• Manual retries.
• Network timeouts.
• Repeated webhook deliveries.
• Queue redelivery.
• Worker restarts.
• Uncertain API responses.

Design the operation so processing the same event twice does not create two products or apply an incorrect result.

Use:

• Stable external identifiers.
• Upsert operations.
• Unique database constraints.
• Processed-event records.
• Idempotency keys.
• Version or timestamp checks.

Idempotency allows a failed request to be retried without repeating the intended side effect.

Message-processing systems may deliver the same message more than once, so background handlers should be able to recognize and safely process repeated messages.

Example: The ERP sends the same product.updated event twice for product 1872. The integration identifies the e-shop product through erp_product_id = 1872 and updates it rather than creating another product.

12. Handle failures and retries

Not every failure should be retried.

Separate failures into categories.

Validation errors

Examples:

• Missing SKU.
• Invalid price.
• Unknown VAT category.

These usually require data correction. Repeating the same request without changing the data will not solve the problem.

Authentication and authorization errors

Examples:

• Expired credentials.
• Invalid API key.
• Missing permission.

Stop or pause the process and notify an administrator.

Temporary failures

Examples:

• Network timeout.
• Service temporarily unavailable.
• HTTP 500, 502, 503, or 504.
• Temporary rate limiting.

Retry temporary failures with:

• A maximum number of attempts.
• Increasing delays.
• Random jitter.
• A final failed state.

Exponential backoff increases the waiting time between retries, while jitter prevents many workers from retrying at exactly the same moment. Retries should always have a maximum limit.

When an API returns Retry-After or rate-limit reset information, respect those values before sending another request.

Example: Updating a product returns HTTP 503. The integration retries after 2 seconds, then 5 seconds, then 12 seconds. After the final failed attempt, it records the product as failed and continues processing the remaining products.

13. Handle partial failures

A synchronization run may update some products successfully while others fail.

Do not treat the complete run as if nothing happened.

Record the outcome of each item:

• Created.
• Updated.
• Skipped.
• Unchanged.
• Invalid.
• Failed.
• Waiting for retry.

Decide whether the run should:

• Continue after an individual product fails.
• Stop immediately.
• Retry only failed products.
• Roll back completed changes when possible.

For product synchronization, continuing with the remaining products is usually more practical than stopping the complete run because one product contains invalid data.

Example: A batch contains 100 products. Ninety-eight synchronize successfully. One has an unknown VAT code, and one API request times out. The integration records the first as invalid, schedules the second for retry, and preserves the 98 successful updates.

14. Define how deletions and inactive products work

Decide what should happen when a product is deleted or disabled in the ERP.

Possible actions include:

• Permanently delete it from the e-shop.
• Hide it from customers.
• Mark it as inactive.
• Set stock to zero.
• Archive it.
• Take no automatic action.

Permanent deletion may also remove:

• Product URLs.
• Images.
• Reviews.
• Order references.
• SEO information.
• Historical data.

In many cases, disabling or archiving a product is safer than deleting it automatically.

Example: When an ERP product becomes inactive, the integration sets the e-shop product status to hidden and its stock to zero. It does not permanently delete the product.

15. Add logging and monitoring

Every synchronization run should produce enough information to understand what happened.

Record:

• Run identifier.
• Trigger type.
• Start time.
• End time.
• Duration.
• Checkpoint.
• Number of records retrieved.
• Number created.
• Number updated.
• Number skipped.
• Number failed.
• Retry attempts.
• General error messages.

For individual failures, record:

• External product identifier.
• Processing step.
• Error category.
• Error message.
• Retry status.

Do not log:

• Passwords.
• API keys.
• Access tokens.
• Complete authentication headers.
• Unnecessary personal information.

Add alerts for important conditions such as:

• Synchronization has not run recently.
• Failure rate exceeds an acceptable level.
• Credentials have expired.
• API rate limits are repeatedly reached.
• A job has exceeded its expected duration.
• The queue contains too many unprocessed items.

Example: A synchronization dashboard shows that run SYNC-20260712-1045 retrieved 300 products, created 12, updated 270, skipped 15, and failed 3. An administrator can inspect the three failed product identifiers and their errors.

16. Test the integration

Test each part separately and then test the complete workflow.

Include:

• Field-mapping tests.
• Validation tests.
• Transformation tests.
• Authentication tests.
• Authorization tests.
• API contract tests.
• Creation tests.
• Update tests.
• Duplicate-delivery tests.
• Retry tests.
• Timeout tests.
• Rate-limit tests.
• Partial-failure tests.
• Large-dataset tests.
• Manual synchronization tests.

Use test or sandbox environments when the connected systems provide them.

Do not test destructive operations directly against production data without safeguards.

Example: Send the same product event twice and verify that only one e-shop product exists. Then simulate an e-shop timeout and confirm that retrying the operation updates the same product instead of creating a duplicate.

17. Deploy and operate the integration

Decide where the integration will run.

Possible environments include:

• The ERP backend.
• The e-shop backend.
• A separate integration service.
• A serverless function.
• A scheduled worker.
• A queue worker.

Prevent incompatible synchronization runs from overlapping.

Possible protections include:

• Distributed locks.
• Unique active-job constraints.
• Queue-level deduplication.
• Run-status checks.

Separate configuration from code.

Configuration may include:

• API endpoints.
• Credentials.
• Batch sizes.
• Timeouts.
• Retry limits.
• Synchronization frequency.
• Field mappings.
• Feature switches.

Example: A scheduler starts product synchronization every 15 minutes. Before starting, the worker checks whether another product synchronization is active. If one is still running, the new execution is skipped or delayed.

18. Document and maintain the integration

Document:

• Integration goal.
• Source and destination systems.
• Source-of-truth decisions.
• Field mappings.
• Authentication method.
• API endpoints.
• Trigger methods.
• Scheduling.
• Batch size.
• Timeout values.
• Retry rules.
• Error handling.
• Deployment process.
• Monitoring.
• Recovery procedures.
• Manual synchronization procedure.
• Known limitations.

APIs and business requirements change over time.

Review the integration when:

• An API version changes.
• A field is renamed.
• Authentication changes.
• Product volume increases.
• New product types are introduced.
• Business ownership of a field changes.
• Synchronization becomes slower.
• Failure rates increase.

Example: The ERP introduces multiple warehouses. The integration documentation must be updated to explain whether the e-shop receives total stock, stock from one warehouse, or stock calculated using a new availability rule.

A reliable API integration is not only a script that transfers data. It is a controlled process with clear ownership, secure execution, predictable failure handling, measurable results, and documentation that allows another developer to understand and maintain it.