Using Pagination in the API to Efficiently Retrieve Large Datasets
When accessing large datasets through the API, it’s essential to manage data retrieval efficiently. Pagination is a technique that divides a large dataset into smaller, manageable chunks or “pages.” This approach enhances performance and prevents overwhelming both the client and server with excessive data in a single request.
Each request retrieves a maximum of 100 records. Use the offset parameter to specify the number of records to skip before starting the return.
| Query | Type | Description |
|---|---|---|
| offset | number | Indicates the number of records to skip |
Example Query Params:
curl --request GET \
--url 'https://api-prod-us-west-2.solinkcloud.com/v2/reports?offset=100' \
--header 'Authorization: Bearer {ACCESS_TOKEN} ' \
--header 'accept: application/json' \
--header 'x-api-key: {API_KEY}'Here offset is set to 0, meaning retrieval starts from the first record.
Pagination in Action
To retrieve the entire dataset, loop through the data by incrementally adjusting the offset based on the set 100 limit until all records are fetched.
First Request: offset= 0 - Retrieves records 1 to 100.
Second Request: offset=100 - Retrieves records 101 to 200.
Subsequent Requests: Continue incrementing the offset by 100 until all records are retrieved.
Pagination Limit Note
List endpoints (such as /v2/events, /v2/cameras, and similar) reject requests where offset ≥ 10,000.
Workaround: If you need to retrieve more than 10,000 records, split your queries into smaller time ranges to stay under the pagination ceiling.
Another suggestion is to implement the following:
When offset approaches the ceiling (e.g., ≥ 9,900) or a page returns fewer than 100 items, advance the time window by setting the next request’s start to the last item’s timestamp plus a small increment (e.g., +1 ms), keep the same end, and reset offset to 0.
Repeat until you reach your overall end time. This pattern prevents hitting the offset ceiling and guarantees full coverage without overlaps or gaps.