Weeks to Hours: Building a Declarative Integration Engine
I built this at Sprinto, a compliance automation platform. But the problem is not specific to compliance. Any product that pulls data from other products will hit it. From here on, everything is generic.
The problem
Our product needed data from hundreds of third party APIs. Employee lists from HR tools, vulnerabilities from scanners, machines from cloud providers. Every integration looked the same from a distance: authenticate, fetch, paginate, transform, store, evaluate. Yet each one took weeks of engineering time, because each one was hand written code with its own bugs, its own retry logic, and its own review cycle.
You can buy your way out with a unified API vendor like Merge.dev or Truto. We tried that. It costs real money (about $200k in contracts for us), you inherit someone else’s data model, and the long tail of APIs your customers actually ask for is never covered. Renting the problem is not solving it.
The bet
If every integration is the same five steps in different clothes, stop writing code. Describe the integration as data, and write one engine that runs the description.
The framing mattered from day one. We were not building an ingestion tool. We were building a declarative workflow executor, and ingestion was just the first job we gave it. A step is a general operation, not a data pulling helper. And because a plan is plain JSON or YAML, it is portable: you can import one, reuse it, or cherry pick steps from another plan.
An integration became a plan: a JSON document with three parts.
- Parameters: typed inputs, like a region or a project id.
- Steps: a list of operations from a small fixed vocabulary. HTTP calls, transforms, loops, polling, CSV parsing, entity mapping, entity writes, rule evaluation.
- Output: what the run produces.
Each step has a name. Any later step can reference an earlier step’s output by that name. Steps declare what they depend on and when they should be skipped. So a plan is really a small dataflow graph, and the engine is a generic interpreter that walks it.
Building one integration this way is slower than hacking it in code. Building the hundredth is an afternoon.
The design clues
These are the decisions that made it work. Most of them are about saying no.
A closed step vocabulary, not a scripting language. The tempting design is to let authors write JavaScript. We refused. Every step is one of a fixed set of types, and each type is validated against a JSON Schema before the plan can be saved. A closed vocabulary means the engine knows the shape of every operation. It can validate plans statically, retry steps safely, and render an editor UI for free. When something is missing, we add a step type once, and every plan gets it.
Small enough to fit in two pages. The whole concept can be explained in a two page doc, and that was a goal, not an accident. We optimized for a few small primitives that are easy to grasp but can do a whole lot. That is what made the engine learnable by any human, and later by any agent.
Three small languages instead of one big one. Plans need three kinds of dynamic behavior, and we kept them separate on purpose. String interpolation (${...}) for URLs and headers, because it is cheap and always safe to evaluate. Plain dot paths for picking a field out of an input. And JSONata, a sandboxed query language, for real transformation logic. The separation means the easy 90 percent stays easy to read, and the hard 10 percent is contained in one place with clear rules. There is no arbitrary code execution anywhere.
Credentials are references, never values. A step does not hold a secret. It holds an alias, like ${credentials.github.token}. The binding from alias to actual credential lives outside the plan. This one rule bought us two big things. The same plan JSON is portable across customers, and plans can be published as templates. A template is just a plan with its credential bindings stripped out. A customer instantiates it and binds their own credentials. Behind the alias sits a secret store, encrypted at rest, that injects values only at run time. On top of it we built an API Playground, a lightweight Postman inside the product, so authors can test calls with bound credentials without ever seeing them. Token refresh and the store’s transaction rules deserve their own article.
The engine can pause and resume. The runner’s execution state (resolved parameters, every step’s output, cursor position) serializes to a plain object and back. That means a long run can checkpoint across worker boundaries and survive restarts instead of holding everything in memory and praying.
Custom data, not just custom fetching. Plans can define their own entity types with typed fields and references. So an integration is not limited to the shapes our product already knew about. A plan can invent a new kind of record, and the product renders list and detail pages for it automatically. Compliance checks are themselves just an evaluation step over those entities. Fetching, modeling, and checking all live in one document.
The payoff we did not plan for
We made these choices years before LLMs could write code you would trust in production. Then LLMs arrived, and the declarative design turned out to be the perfect target for them.
An AI cannot safely write and deploy integration code. But it can compose validated lego pieces. Every step type has a schema, so a generated plan is either valid or rejected before it runs. The blast radius of a bad generation is a failed validation, not a security incident.
So we put an AI editor in front of the engine, and customers now build their own integrations by describing what they want. One design detail here matters. The AI never edits the live plan. It works on a shadow copy carried alongside the real one, with a version number. If the plan changed underneath the shadow, the state machine flags a conflict and the user must reconcile. Promoting a shadow to live snapshots the old plan into a log first, so there is always an undo. It is optimistic concurrency control, applied to letting an AI touch things people care about.
The business result: when a prospect asks for an integration we do not have, the answer is hours, not a roadmap quarter. That closed six figure deals against bigger competitors like Vanta and Drata.
What it cost
Honesty section, because every design has a bill.
- The surprise: the engine itself was not the slow part. We built the core in about four weeks with AI assisted engineering. We did the design and planning, and an AI agent executed. The bill is not in building the engine. It is in everything around it.
- Declarative systems need their own tooling. Debugging a failing plan is not like debugging code. You need run logs per step, input and output inspection, and dry runs. We had to build all of that. But it was a good tradeoff: none of that tooling is business specific. Verbose logging, dry runs, and performance traces are built once at the engine level, and every workflow on it gets them for free.
- A closed vocabulary means a queue of “the engine can’t do X yet” requests. Each one is a deliberate decision: add a step type, extend one, or say no.
- Expression languages hide performance cliffs. We once found a query pattern in JSONata that went super quadratic and froze workers for 15 minutes. That story, and the fix, is its own article.
If you are building one
Start with the smallest step vocabulary that covers one real integration end to end. Validate everything at save time, not run time. Keep secrets out of the documents from day one, because retrofitting that is miserable. And write the interpreter so its state can serialize. You will want checkpoints long before you expect to.
The deeper lesson: constraints compound. Every “no” we said early (no scripting, no inline secrets, no schema-less steps) looked like lost flexibility at the time. Each one later became the exact property that made templates, portability, and AI generation possible.