How Foundry Actually Stores Data: ELT (Not ETL), and OSv2

A different starting premise than traditional BI tools

Traditional BI tools are usually ETL (Extract-Transform-Load). They clean and reshape raw data as they bring it in, and only store the already-polished result. That’s a habit from an era when storage was expensive and limited.

Foundry is closer to ELT (Extract-Load-Transform). It brings in and stores raw data as close to its original form as possible first, and cleaning/transformation happens as a separate later step (the pipeline). This is possible because cloud-based distributed storage has drastically cut the cost burden of keeping raw data around.

The practical payoff of this ordering: if you ever find yourself wanting to “go check that original raw data again,” the original is still there. Even if you discover a bug in your cleaning logic later, you can just reprocess from the source.

Where does data live before it reaches the ontology?

Before data is mapped into the Ontology, it exists in Foundry in a form called a Dataset. If you compare it to other systems, a Dataset plays the same role as the staging table / raw table that holds unmodified source data in an ETL pipeline. The closest way to think about it: “a container that holds unrefined raw data.”

The implementation, though, is different from a typical table. A Dataset is fundamentally a wrapper around a set of files. If it’s structured (tabular) data, it’s stored internally in an open-source file format like Parquet, with its column structure managed alongside as a separate schema. From the outside, you can treat it like a familiar “table” with columns, rows, and a schema — but underneath, it’s really a cluster of files distributed across a file system. That’s the key thing to understand.

A Dataset is managed through three concepts:

• Branch: Nearly identical to a Git branch. A branch is just a pointer to the most recent transaction on that branch. Most work starts on a single master branch, with child branches created as needed. (Unlike Git, though, merging between branches isn’t supported.)
• Transaction: Similar to a Git commit. Every change to a Dataset accumulates as one transaction, and the sequence of these transactions is the Dataset’s version history.
• Transaction types: SNAPSHOT (rewrite everything), APPEND (add only), UPDATE, DELETE, and others. DELETE doesn’t actually remove the file on the spot — it marks it as “hidden from future queries.” Actually removing it from storage requires a separate retention policy.

Because of this structure, a Dataset comes with things a plain table doesn’t have by default — permission management, schema management, and version control (you can query data as of a past point on a given branch) are all built into the Dataset itself. Understanding it as “looks like a table, but is version-controlled like Git as a bundle of files” will save you confusion later when designing pipelines.

Being file-based is both an advantage and a constraint. It’s not a structure where you do row-level, on-the-spot INSERT/UPDATE/DELETE like an RDB. Data accumulates transaction by transaction — either “this time, rewrite the whole thing” (SNAPSHOT) or “this time, only what was added” (APPEND/incremental).

The ontology’s actual storage — OSv2

Once a refined Dataset is mapped to an Object Type in the Ontology, that data is managed through Palantir’s own storage mechanism: OSv2 (Object Storage V2). OSv1 (internally called Phonograph) was used previously, but OSv1 was discontinued as of June 30, 2026, and OSv2 is now the standard storage layer for the ontology.

OSv2 stores ontology data in a form optimized for fast indexing and search (Search Around) and for user edits (writeback). In the OSv1 era, enabling user edits required building a separate writeback dataset. In OSv2, edits made through Actions are reflected immediately, without that extra step.

Summary

If you try to treat Foundry like a SQL-based RDB, this is where things keep not lining up. Pipeline design starts to feel natural once you stop thinking “fix one row on the spot” and start thinking “rewrite the whole thing, or layer on just the changes.”