How Foundry Actually Stores Data: ELT (Not ETL), and OSv2
A different starting premise than traditional BI tools
Traditional BI tools are usually ETL (Extract-Transform-Load). They clean and reshape raw data as they bring it in, and only store the already-polished result. That’s a habit from an era when storage was expensive and limited.
Foundry is closer to ELT (Extract-Load-Transform). It brings in and stores raw data as close to its original form as possible first, and cleaning/transformation happens as a separate later step (the pipeline). This is possible because cloud-based distributed storage has drastically cut the cost burden of keeping raw data around.
The practical payoff of this ordering: if you ever find yourself wanting to “go check that original raw data again,” the original is still there. Even if you discover a bug in your cleaning logic later, you can just reprocess from the source.
Where does data live before it reaches the ontology?
Before data is mapped into the Ontology, it exists in Foundry in a form called a Dataset. If you compare it to other systems, a Dataset plays the same role as the staging table / raw table that holds unmodified source data in an ETL pipeline. The closest way to think about it: “a container that holds unrefined raw data.”
The implementation, though, is different from a typical table. A Dataset is fundamentally a wrapper around a set of files. If it’s structured (tabular) data, it’s stored internally in an open-source file format like Parquet, with its column structure managed alongside as a separate schema. From the outside, you can treat it like a familiar “table” with columns, rows, and a schema — but underneath, it’s really a cluster of files distributed across a file system. That’s the key thing to understand.
A Dataset is managed through three concepts:
- Branch: Nearly identical to a Git branch. A branch is just a pointer to the most recent transaction on that branch. Most work starts on a single
masterbranch, with child branches created as needed. (Unlike Git, though, merging between branches isn’t supported.) - Transaction: Similar to a Git commit. Every change to a Dataset accumulates as one transaction, and the sequence of these transactions is the Dataset’s version history.
- Transaction types: SNAPSHOT (rewrite everything), APPEND (add only), UPDATE, DELETE, and others. DELETE doesn’t actually remove the file on the spot — it marks it as “hidden from future queries.” Actually removing it from storage requires a separate retention policy.
Because of this structure, a Dataset comes with things a plain table doesn’t have by default — permission management, schema management, and version control (you can query data as of a past point on a given branch) are all built into the Dataset itself. Understanding it as “looks like a table, but is version-controlled like Git as a bundle of files” will save you confusion later when designing pipelines.
Being file-based is both an advantage and a constraint. It’s not a structure where you do row-level, on-the-spot INSERT/UPDATE/DELETE like an RDB. Data accumulates transaction by transaction — either “this time, rewrite the whole thing” (SNAPSHOT) or “this time, only what was added” (APPEND/incremental).
The ontology’s actual storage — OSv2
Once a refined Dataset is mapped to an Object Type in the Ontology, that data is managed through Palantir’s own storage mechanism: OSv2 (Object Storage V2). OSv1 (internally called Phonograph) was used previously, but OSv1 was discontinued as of June 30, 2026, and OSv2 is now the standard storage layer for the ontology.
OSv2 stores ontology data in a form optimized for fast indexing and search (Search Around) and for user edits (writeback). In the OSv1 era, enabling user edits required building a separate writeback dataset. In OSv2, edits made through Actions are reflected immediately, without that extra step.
Worked Example: Fixing a Customer Tier Field
In an RDB, UPDATE customers SET tier='VIP' WHERE id=42 would be a one-liner. In Foundry the flow looks like this:
- Find the source dataset: use Lineage to see which upstream dataset the tier value is loaded into and at which pipeline stage it is computed.
- Decide whether to change the logic: if the tier logic itself is wrong, fix the transform in the Code Repository. If one record needs a special exception, a common pattern is a separate reference dataset (an allow-list style table).
- Run a transaction: re-running the pipeline creates a new transaction (APPEND or SNAPSHOT) and the previous state stays in history.
- Verify in OSv2: once mapped to ontology objects, confirm in Ontology Manager that the property value has been updated.
It looks like more steps than an instant RDB update, but in exchange “when and why did this value change” is recorded automatically in transaction history and Lineage.
RDB vs. Foundry Storage
| Aspect | Traditional RDB | Foundry |
|---|---|---|
| Processing order | ETL (transform, then load) | ELT (load, then transform) |
| Single-row edit | Immediate via UPDATE | Re-run pipeline, new transaction created |
| Change history | Lost without a separate audit table | Always preserved per transaction |
| Deletion | Physical delete | Logical delete (recorded in the next transaction) |
| Rollback | Requires restoring a backup | Switch to an earlier transaction/branch |
Summary
| Stage | Storage form | Characteristics |
|---|---|---|
| Raw ingestion | Dataset (file-based, e.g. Parquet) | Plays a role similar to a staging/raw table in other systems. ELT-style, version-controlled via Branch and Transaction, only SNAPSHOT/APPEND-style transactions |
| After ontology mapping | OSv2 | Fast indexing/search, edits via Actions reflected immediately |
If you try to treat Foundry like a SQL-based RDB, this is where things keep not lining up. Pipeline design starts to feel natural once you stop thinking “fix one row on the spot” and start thinking “rewrite the whole thing, or layer on just the changes.”
Six checks before ‘fixing’ data in Foundry
- Have you distinguished whether the data is a Dataset (before ontology mapping) or an ontology object?
- Have you considered correcting in a pipeline transform instead of altering the source?
- Have you chosen the Transaction type to match whether you rewrite everything (SNAPSHOT) or add to it (APPEND)?
- Since deletion is logical, have you checked it against retention and audit requirements?
- Do you validate on a Branch first and then merge?
- Have you checked what downstream consumers reference the data you are changing?
Quick answers
Q. Why can’t I just UPDATE a single row?
A Dataset is internally a bundle of files and tracks change history per Transaction. Rebuilding through a transform fits the design better than instant row-level edits.
Q. Does deleting remove the data immediately?
DELETE is logical: it records the fact of deletion in the next Transaction. Check physical removal and retention policy separately.
Q. Does traditional BI experience get in the way?
Applying ETL habits as-is will cause friction, but once you understand the ELT order — keep the source, then transform — history tracing and reproducibility become advantages.
One line to remember
In Foundry, think not of ‘fixing a row’ but of ‘rebuilding the correct result through a transform.’ Then Transactions, Branches, and logical deletion all look like different expressions of the same philosophy.