Make a new dataset from a query, under the same rules: a Munitas walkthrough

How to read this page

Every step names the tool or the part of Munitas where it happens. The label under the title says where, and the step reads in the same order each time: what is happening, the evidence, and then what to look for in it.
Console steps happen in the Munitas console, the web app that people open in a browser. They show real screenshots, and clicking one enlarges it.
Python steps happen in Python, on the person’s own computer, in a notebook or in a terminal. Python is a window where a person types a line and sees the answer straight away. From there the person uses the Munitas client library, which sends requests to Munitas, and DuckDB, a free tool for analysing tables. Each dark box on the page is one of these steps.
In a dark box, the top part is exactly what was typed and the bottom part is exactly what came back. me is the client library signed in as the person named on the label, and custodian is the client library signed in as the custodian.
Every step was run for real by a script that types each one, so the answers are genuine and nothing has been edited. Every name of a person, patient or card holder in the example data is invented.

Words used on this page

Organisation: A company or hospital group that uses Munitas. Each organisation sees only its own data.
Department: A team inside an organisation that owns some of its data, such as Cardiology.
Dataset: A named collection of records, such as a table of patient admissions.
Custodian: A person who holds the data custodian role, which allows deciding who may read data. A custodian decides for the departments they are an approver for.
Access level: How restricted a dataset is. Raw is the most restricted, and nobody may read it without permission. Published is the most open.
Lease: Permission, given by the custodian, to read one dataset for one stated purpose and for a limited time.
Query: A question about a dataset, written in a language called SQL.
Sensitivity label: How identifying a column is. none does not identify anybody, quasi is a detail that could help identify a person when combined with others, and phi is protected health information.
Token: A short-lived password that also states why the person wants to read.
Sealed: Finished and closed for good, so that it can never be edited.
Iceberg table: An open way of storing a table that tools such as DuckDB can read directly.
Version: One sealed state of a dataset. Changing the contents means making a new version, never editing an old one.
Container: An isolated environment, with no network access of its own, where Munitas runs an agent.

PART ONE

Making a new dataset from a query

Sam, a researcher, only needs the older patients with a long-term heart condition. Munitas lets a person make a new dataset by writing a query, which is a question about tables written in a language called SQL, over datasets the person may already read. Munitas runs the query itself, not the person's computer, and holds the new dataset to the same rules as the data it came from. Sam holds a lease, which is permission from the custodian to read one dataset for a stated purpose and a limited time, and it is still in force.

Sam

Munitas shows its plan before anything runs

Python on Sam’s own computer, using the Munitas client library

Sam writes a query that joins the restricted admissions with the public diagnosis lookup and keeps patients over 65 with a chronic condition, then asks Munitas to plan it. Nothing runs yet. Munitas replies with the columns the new dataset would have, and the sensitivity label each one carries: none for a column that does not identify anybody on its own, quasi for a detail that could help identify a person when combined with others, and phi for protected health information, which means facts about a person's health.

Typed by Sam, in Python on Sam’s own computerme is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Sam.

sql = """SELECT a.admission_id, a.age, a.diagnosis_code, d.description, d.chronic,
       a.length_of_stay_days, a.readmitted_30d
FROM a JOIN d ON d.code = a.diagnosis_code
WHERE a.age > 65 AND d.chronic"""
draft = me.draft(inputs=[{"dataset": "admissions", "alias": "a"}, {"dataset": "diagnosis_codes", "alias": "d"}], sql=sql, name="chronic-heart-patients-over-65-1012-1259",
                 primary_key=["admission_id"], purpose="readmission study")
for f in draft["fields"]:
    print(f"{f['name']:<22}{f['type']:<8}{f['sensitivity']}")
print("The new dataset would be class", draft["output_class"])

What came back

admission_id          string  none
age                   int     none
diagnosis_code        string  none
description           string  none
chronic               bool    none
length_of_stay_days   int     none
readmitted_30d        bool    none
The new dataset would be class RAW

All seven columns are none, as the contract that Devi uploaded said for each of them. The last line says the new dataset would be RAW: a new dataset takes the strictest access level among its inputs, and the admissions are Raw.

Sam

Sam confirms, and Munitas runs the query itself

Python on Sam’s own computer, using the Munitas client library

After Sam confirms, Munitas copies the input files into a sandbox, an isolated container with no network connection and no passwords. The query runs there, and Munitas checks every row of the result against the plan. The query never runs on Sam's computer. Sam's session only waits for the answer.

Typed by Sam, in Python on Sam’s own computerme is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Sam. To run the query, Munitas starts it in a sandbox, an isolated container with no network, and writes the result back as a new dataset.Names used here that are not defined in this box: draft was set in step 1.

confirmed = me.confirm(draft)
result = me.wait(confirmed["id"])
print("Status:", result["status"])
print("New dataset:", result["target_name"], "|", result["output_class"])

What came back

Status: succeeded
New dataset: chronic-heart-patients-over-65-1012-1259 | RAW

The status is succeeded, and the new dataset is RAW again. Sam did not choose that level: it follows from the datasets the query read.

Sam

The new dataset appears, owned by Cardiology

Munitas console · Datasets · /datasets

The new dataset belongs to the same department as the data it came from, so Hartley, the custodian of that department, is still the person who decides who may read it. Sam made it, so Sam can read it straight away without asking anybody.

Screenshot: The new dataset appears, owned by Cardiology

The line for the new dataset says 1 of 1 you can read, and the access level is still Raw. The name ends in the month, day, hour and minute when it was made, so that a second run never meets a name that is already taken.

Sam

The new dataset opens like any other table

Python on Sam’s own computer, using the Munitas client library and DuckDB

Munitas sealed the result, which means it is finished and closed for good, so it can never be edited. It is also an Iceberg table, an open way of storing a table that tools such as DuckDB read directly. Sam queries it.

Typed by Sam, in Python on Sam’s own computerme is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Sam. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.Names used here that are not defined in this box: token is the token requested at the start of the same session.

con = me.attach_duckdb(token)
print(con.sql("""SELECT admission_id, age, description, length_of_stay_days, readmitted_30d FROM lake."chronic-heart-patients-over-65-1012-1259".v1 ORDER BY age DESC LIMIT 5"""))
print(con.sql("""SELECT count(*) AS patients, round(avg(age), 1) AS mean_age, round(avg(length_of_stay_days), 1) AS mean_stay_days FROM lake."chronic-heart-patients-over-65-1012-1259".v1"""))

What came back

┌──────────────┬───────┬──────────────────────────────────┬─────────────────────┬────────────────┐
│ admission_id │  age  │           description            │ length_of_stay_days │ readmitted_30d │
│   varchar    │ int64 │             varchar              │        int64        │    boolean     │
├──────────────┼───────┼──────────────────────────────────┼─────────────────────┼────────────────┤
│ ADM-0383     │    96 │ Coronary artery disease          │                   7 │ true           │
│ ADM-0172     │    96 │ Atrial fibrillation, unspecified │                   2 │ false          │
│ ADM-0038     │    96 │ Heart failure, unspecified       │                   6 │ false          │
│ ADM-0220     │    95 │ Atrial fibrillation, unspecified │                   7 │ true           │
│ ADM-0286     │    95 │ Essential hypertension           │                   4 │ true           │
└──────────────┴───────┴──────────────────────────────────┴─────────────────────┴────────────────┘

┌──────────┬──────────┬────────────────┐
│ patients │ mean_age │ mean_stay_days │
│  int64   │  double  │     double     │
├──────────┼──────────┼────────────────┤
│      106 │     77.2 │            4.8 │
└──────────┴──────────┴────────────────┘

The second answer counts the patients in the new dataset and gives their average age, which is above 65 as the query required.

Sam

The console records where the new dataset came from

Munitas console · A dataset version · /versions/<id>

Every version of a dataset records what produced it: the step, who ran it, and the exact versions of the datasets it was made from.

Screenshot: The console records where the new dataset came from

Your access reads You can read this until a date, for the stated purpose. Where this came from shows the step derive and two Made from versions, one for the admissions and one for the diagnosis lookup.

Sam

The query is kept exactly as written

Python on Sam’s own computer, using the Munitas client library

Anybody auditing the new dataset can follow it back. Munitas keeps the purpose, the datasets and versions it was made from, and the text of the query.

Typed by Sam, in Python on Sam’s own computerme is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Sam.Names used here that are not defined in this box: result was set in step 2.

record = me.derivation(result["id"])
print("Purpose :", record["purpose"])
print("Inputs  :", ", ".join(f"{i['dataset']} version {i['version']} ({i['class']})" for i in record["inputs"]))
print("Query   :")
print(record["sql"])

What came back

Purpose : readmission study
Inputs  : admissions version 1 (RAW), diagnosis_codes version 1 (PUBLISHED)
Query   :
SELECT a.admission_id, a.age, a.diagnosis_code, d.description, d.chronic,
       a.length_of_stay_days, a.readmitted_30d
FROM a JOIN d ON d.code = a.diagnosis_code
WHERE a.age > 65 AND d.chronic

The inputs line names admissions version 1 (RAW) and diagnosis_codes version 1 (PUBLISHED), which are the most restricted and the most open access levels. The query appears word for word.

PART TWO

The rules still apply to the new dataset

A new dataset is only safe if nobody can use it to get around the rules. This part tests that with a colleague, with a query that reaches too far, and with the custodian taking the access back.

Devi

A colleague without access is refused

Python on Devi’s own computer, using the Munitas client library and DuckDB

Devi is a data engineer in the same organisation, with no lease on the new dataset. The new dataset is as restricted as the data it came from, so Devi cannot open it.

Typed by Devi, in Python on Devi’s own computercolleague is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Devi. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.

colleague_token = colleague.catalog_token(purpose="curiosity")
other = colleague.attach_duckdb(colleague_token)
try:
    print(other.sql('SELECT count(*) FROM lake."chronic-heart-patients-over-65-1012-1259".v1'))
except Exception as exc:
    print("Refused:", exc)

What came back

Refused: HTTP Error: GetTableInformation endpoint returned response code Forbidden_403 with message "no role reaches class RAW"

The reason is the same as before: no role reaches class RAW. Working in the same organisation is not enough.

Sam

A query that reads outside its inputs is stopped

Python on Sam’s own computer, using the Munitas client library

A query may read the datasets it names and nothing else. Sam tries one that reads a file from the machine instead.

try:
    me.draft(inputs=[{"dataset": "admissions", "alias": "a"}], sql="SELECT content AS admission_id FROM read_text('/etc/passwd')",
             name="chronic-heart-patients-over-65-1012-1259-escape", primary_key=["admission_id"], purpose="readmission study")
except MunitasError as exc:
    print("Refused:", exc)

What came back

Refused: HTTP 422: that query reaches outside the datasets it declared (a file or an address). A query may read only its declared inputs

Munitas refuses with that query reaches outside the datasets it declared. Nothing ran.

Hartley

Every refusal is on record, with the reason

Munitas console · Access decisions · /audit

The decision log lists every request to read data. Each request is recorded twice: once for whether it was allowed, and once for whether access was actually given. Here the log is filtered to the requests that were refused.

Screenshot: Every refusal is on record, with the reason

Every row says no. The two newest are Devi's attempt, with the stated purpose curiosity, and Sam's first attempt, readmission study. Both give the reason No role reaches class RAW.

Hartley

Hartley withdraws access, and the new dataset closes too

Python on Hartley’s and Sam’s own computers, using the Munitas client library and DuckDB

Hartley ends the lease. Sam immediately loses the admissions table, and the new dataset too, because Sam's access to the new dataset rested on the same lease.

Typed by Hartley, in Python on Hartley’s own computercustodian is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Hartley.Names used here that are not defined in this box: lease is the lease the custodian approved earlier in the same session.

custodian.revoke(lease)
print("The lease is withdrawn.")

What came back

The lease is withdrawn.

time.sleep(3)
con = me.attach_duckdb(token)
for name in ["admissions", "chronic-heart-patients-over-65-1012-1259"]:
    try:
        print(con.sql(f'SELECT count(*) FROM lake."{name}".v1'))
    except Exception as exc:
        print(f"{name}: refused, {str(exc).split('message ')[-1]}")

What came back

admissions: refused, "lease 881ec05f-a15b-4772-b131-bb94e674e16b revoked; no role reaches class RAW"
chronic-heart-patients-over-65-1012-1259: refused, "lease 692021a3-f684-47b1-9fd8-503edebfdc5d revoked; no role reaches class RAW"

Both tables report refused, and the reasons name the revoked leases.

Sam

The console shows the new dataset closed again

Munitas console · Datasets · /datasets

The new dataset is still listed, so Sam can still know that it exists, but Sam can no longer read it.

Screenshot: The console shows the new dataset closed again

The line for the new dataset now says 0 of 1 you can read, where it said 1 of 1 earlier.