# Consuming Mechbase data

*Keeping a warehouse, lake, or BI model in lockstep with one installation.*

You're a system integrator, and you want an external store that always matches what's in Mechbase — without re-downloading the whole plant every night. This chapter is the integration playbook: how to do the first full load, how to pull only what changed after that, and how to stay honest when rows get deleted. It's all built on the same **read-only** analytics feeds an analyst uses — no new endpoints, no write access.

For the full endpoint list, parameters, filters, and field meanings, keep [Analyst: Analytics & BI](analytics-bi.md) open alongside this chapter, and the interactive API docs at [`app.mechbase.io/api/docs`](https://app.mechbase.io/api/docs). This chapter assumes those and focuses on the sync patterns.

## Choose your path

Mechbase exposes the same three datasets — **measurements**, **events**, **inspections** — three ways. Pick the path that matches your destination:

- **Live feeds** — paged JSON or CSV you walk page by page. This is your delta channel; it's the only path that takes `updated_since`, so it's how you pull just-the-changes on a schedule.
- **Bulk export** — a whole dataset built as one CSV or Parquet file behind a signed link. This is your load channel; use it for the first full load and for the periodic full refresh.
- **OData feed** — one URL you point Power BI or Excel at. If your "external store" *is* a BI tool that speaks OData, you may not need a pipeline at all — the tool does the paging and refresh for you. See [Analyst: Analytics & BI](analytics-bi.md#odata-details-filtering-and-sorting).

For a warehouse or lake, the pattern that works is a combination:

| Job | Path |
|---|---|
| First load of a dataset into a fresh table | **Bulk export** |
| Every incremental run after that | **Live feed, `updated_since` mode** |
| Periodic full refresh to catch deletes | **Bulk export** again |

> **Note:** everything is scoped to **one installation**. If you sync several sites, run the whole pattern once per installation ID.

## What you'll need

Two things, both covered in detail in [Analyst: Analytics & BI](analytics-bi.md#before-you-start-two-things-you-need):

- **Your installation ID** — the numeric ID in every URL. The base for everything here is `https://app.mechbase.io/api/installations/<INSTALLATION_ID>/analytics/` (swap the host for a self-hosted Mechbase).
- **An API token**, sent as `Authorization: Bearer <TOKEN>` on every request.

One integrator-specific tip: ask an administrator for a **service token**, not an interactive one. It raises the bulk-export row cap from 100,000 to 10,000,000 — the difference between "a demo" and "the real plant".

> **Heads-up:** the bulk-export **download link is pre-signed** — the signature in the URL is the authorization, so don't send the Bearer header when you fetch the file. Everything else needs the header.

## The first full load

Seed each destination table from a bulk export. It's a three-step flow — start the job, poll it, download the signed link:

```python
import requests, time, pandas as pd

BASE = "https://app.mechbase.io/api/installations/<INSTALLATION_ID>/analytics"
HEADERS = {"Authorization": "Bearer <TOKEN>"}

def bulk_export(dataset, fmt="parquet", filters=None):
    job = requests.post(f"{BASE}/exports/", headers=HEADERS, json={
        "dataset": dataset,            # measurements | events | inspections
        "format": fmt,                 # parquet | csv
        "filters": filters or {},
    }).json()
    while True:
        s = requests.get(job["status_url"], headers=HEADERS).json()
        if s["state"] in ("done", "failed", "expired"):
            break
        time.sleep(3)
    if s["state"] != "done":
        raise RuntimeError(s["state"])
    read = pd.read_csv if fmt == "csv" else pd.read_parquet
    return read(s["download_url"])              # signed link — no header
```

Run it once per dataset and write each result to its table:

```python
for dataset in ("measurements", "events", "inspections"):
    df = bulk_export(dataset)
    # ... load df into <dataset> table (create-or-replace) ...
```

Things to plan for, all detailed in [the export reference](analytics-bi.md#bulk-export-a-whole-dataset-as-one-file):

- **Row caps** — 100,000 on an interactive token, 10,000,000 on a service token.
- **Link lifetime** — the signed `download_url` is short-lived (~15 minutes). If it expires, poll the status again; each poll mints a fresh link. The built file itself lives 24 hours.
- **Concurrency** — up to 3 export jobs per installation at once; a 4th gets a `429`.

## Keeping in sync

After the first load, don't re-export the world every run. Switch to the live feed's **incremental mode** and pull only what changed since last time by passing `?updated_since=<ISO-8601 timestamp>`.

Two properties of this mode drive the whole design:

- It walks a **last-updated watermark** (`synced_at`), oldest-changed-first. A reading that was backdated or re-finalised after your last run is re-emitted, so you never miss an edit. (Events never change once written, so the events feed walks `timestamp` instead, and re-emits rows exactly on the boundary — upsert and they're harmless.)
- It's **at-least-once** — the same row can come back across runs. That's fine, because you're going to **upsert on the row's stable key**, never blind-insert:
  - measurements → `measurement_uuid`
  - events → `event_id`
  - inspections → `response_uuid`

The loop is: remember when you last ran, pull everything changed since, upsert it, then remember now.

```python
# Walk the paged feed (see 15-cookbook-analytics-bi.md#python--pandas for pull_all).
run_started = "2026-07-01T00:00:00Z"          # the watermark from your last run
df = pull_all("measurements", updated_since=run_started)
# upsert df into your warehouse keyed on measurement_uuid, e.g.
#   MERGE INTO measurements t USING staged s
#     ON t.measurement_uuid = s.measurement_uuid
#     WHEN MATCHED THEN UPDATE SET ...
#     WHEN NOT MATCHED THEN INSERT ...
# then persist "now" as the next run's watermark
```

> **Heads-up:** for a changed-since sync, leave `created_from` / `created_to` **unset**. The created window also constrains an incremental pull, so a late-edited row whose reading time falls outside the window won't re-emit — and you'll silently miss the edit.

## When rows disappear

The feeds only ever **add or update** — a **hard-delete in Mechbase is never announced**. Your incremental job will happily keep a row that no longer exists.

There's no delete feed to subscribe to, so reconcile on a cadence instead: run a **periodic full refresh** — a fresh bulk export loaded into a new table, then swapped in — on whatever interval your accuracy needs (nightly, weekly). The refresh replaces the whole table, so anything deleted upstream simply isn't in the new copy.

Keep the cheap incremental job on its frequent schedule for freshness, and let the full refresh be the slower correctness backstop.

## Scheduling and idempotency

A few rules keep the pipeline safe to re-run and easy to reason about:

- **Advance the watermark only after a run commits.** Store the timestamp you *started* the run, and write it back only once the upsert has landed. If a run crashes mid-way, the next run re-pulls the same window — harmless, because you upsert.
- **Upserts make retries free.** At-least-once delivery plus a `MERGE` on the `*_uuid` / `event_id` key means running the same window twice produces the same table. That's your idempotency guarantee — lean on it instead of trying to pull each row exactly once.
- **Don't persist a cursor between runs.** A cursor is tied to the mode and dataset it was minted for. Start each incremental run fresh from the watermark and walk `next_cursor` to `null` within that run — see [cursor pagination](analytics-bi.md#cursor-pagination).
- **Stagger against the export limit.** Only 3 exports run per installation at once. If your full refresh fans out across datasets or sites, cap concurrency at 3 and back off on a `429`.
- **Download promptly.** The signed link is good for ~15 minutes; fetch the file as soon as the state is `done`, or re-poll for a fresh link.

> **Tip:** the feeds are plain HTTP and the loop above is all you need — the **Mechbase SDK** doesn't wrap them. Reach for the SDK for the operational API instead: one reading by its number (`inst.measurements.get(point_id, seq)`), a point's history (`inst.measurements.iter_for_point(...)`), or a reading's spectra. See [the SDK section](analytics-bi.md#prefer-a-client-library-the-mechbase-sdk).

## Syncing findings into a CMMS

When an analyst records a **finding** on a reading in Mechbase — the fault, its severity, the recommended action, and a due date ([Measurements, trends, and alarms](../measurements.md#recording-a-finding)) — a maintenance system can pick it up as a work request and report back. The findings feed follows the same incremental pattern as the rest of this chapter:

```
GET  /api/installations/<ID>/findings/?since=<ISO-8601>
GET  /api/installations/<ID>/findings/<UUID>/
POST /api/installations/<ID>/findings/<UUID>/status/
```

- **Pull** with `?since=` the time of your last successful run: you get every finding created or changed at or after it, oldest first. Dedupe on `uuid`, and walk `next_cursor` as usual. Narrow with `status=open|acknowledged|closed`, `asset_id` / `asset_external_id`, or `point_id` / `point_external_id`.
- Each finding carries `fault_type`, `severity`, `recommended_action`, `due_by`, the spectral `evidence`, a `web_url` back to the reading, and its `work_order_ref`.
- **Report back** by posting a status: `{"status": "acknowledged", "work_order_ref": "WO-1234"}` when you raise the work order (the reference is required), then `{"status": "closed"}` when it's done. Findings only move forward — open → acknowledged → closed, or straight to closed — and re-sending the same status is harmless. Mechbase shows the state and work-order number on the finding.

## Quick reference

| Job | Path | Key detail |
|---|---|---|
| First full load | Bulk export | POST `/exports/` → poll `status_url` → read pre-signed `download_url` |
| Incremental delta | Live feed | `?updated_since=<ISO-8601>`; walk `next_cursor` to `null` |
| Catch deletes | Bulk export | Periodic full refresh, swap the table |
| Dedupe / upsert keys | — | `measurement_uuid` · `event_id` · `response_uuid` |
| Watermark column | — | `synced_at` (last updated in Mechbase); `timestamp` for events |
| CMMS findings | Findings feed | `GET /findings/?since=` · `POST /findings/<uuid>/status/` |
| Delivery guarantee | Live feed | At-least-once — always upsert, never blind-insert |
| Export row cap | Bulk export | 100k (interactive) / 10M (service token) |
| Signed link TTL | Bulk export | ~15 min (re-poll for a fresh one); file lives 24h |
| Max concurrent exports | Bulk export | 3 per installation, else `429` |
| Full endpoint & field reference | — | [Analyst: Analytics & BI](analytics-bi.md) · [`app.mechbase.io/api/docs`](https://app.mechbase.io/api/docs) |
