Find a table, ask to read it, and the custodian decides: a Munitas walkthrough

How to read this page

Every step names the tool or the part of Munitas where it happens. The label under the title says where, and the step reads in the same order each time: what is happening, the evidence, and then what to look for in it.
Console steps happen in the Munitas console, the web app that people open in a browser. They show real screenshots, and clicking one enlarges it.
Python steps happen in Python, on the person’s own computer, in a notebook or in a terminal. Python is a window where a person types a line and sees the answer straight away. From there the person uses the Munitas client library, which sends requests to Munitas, and DuckDB, a free tool for analysing tables. Each dark box on the page is one of these steps.
In a dark box, the top part is exactly what was typed and the bottom part is exactly what came back. me is the client library signed in as the person named on the label, and custodian is the client library signed in as the custodian.
Every step was run for real by a script that types each one, so the answers are genuine and nothing has been edited. Every name of a person, patient or card holder in the example data is invented.

Words used on this page

Organisation: A company or hospital group that uses Munitas. Each organisation sees only its own data.
Department: A team inside an organisation that owns some of its data, such as Cardiology.
Dataset: A named collection of records, such as a table of patient admissions.
Custodian: A person who holds the data custodian role, which allows deciding who may read data. A custodian decides for the departments they are an approver for.
Access level: How restricted a dataset is. Raw is the most restricted, and nobody may read it without permission. Published is the most open.
Lease: Permission, given by the custodian, to read one dataset for one stated purpose and for a limited time.
Query: A question about a dataset, written in a language called SQL.
Sensitivity label: How identifying a column is. none does not identify anybody, quasi is a detail that could help identify a person when combined with others, and phi is protected health information.
Token: A short-lived password that also states why the person wants to read.
Sealed: Finished and closed for good, so that it can never be edited.
Iceberg table: An open way of storing a table that tools such as DuckDB can read directly.
Version: One sealed state of a dataset. Changing the contents means making a new version, never editing an old one.

PART ONE

A closed table, and a request for access

The Cardiology department of a hospital group owns a table of hospital admissions that a data engineer, Devi, brought in and sealed, and nobody can read it yet. A researcher wants to study it. This part shows the researcher being stopped, asking for access, and being answered by a different person.

Sam

Sam connects a tool and sees one open table

Python on Sam’s own computer, using the Munitas client library and DuckDB

Sam is a researcher in the Health organisation and wants to study readmissions, which are patients who come back to hospital soon after leaving. To start, Sam asks Munitas for a token, which is a short-lived password that also says why Sam wants to read, here a readmission study. Sam then connects DuckDB, a free tool for analysing tables, to the list of tables that Munitas offers, called the catalog, and asks what is in it. DuckDB runs inside Sam's own Python session and reads the tables from Munitas.

Typed by Sam, in Python on Sam’s own computerme is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Sam. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.

token = me.catalog_token(purpose="readmission study")   # shown once, never printed
import duckdb
con = duckdb.connect()
con.execute("INSTALL iceberg; LOAD iceberg;")
con.execute(f"ATTACH 'health' AS lake (TYPE ICEBERG, ENDPOINT 'http://localhost:8000/iceberg', TOKEN '{token}')")
print(con.sql("SELECT schema AS dataset, name AS version FROM (SHOW ALL TABLES) ORDER BY 1"))

What came back

┌─────────────────┬─────────┐
│     dataset     │ version │
│     varchar     │ varchar │
├─────────────────┼─────────┤
│ diagnosis_codes │ v1      │
└─────────────────┴─────────┘

Only one dataset is listed, diagnosis_codes, the public lookup that Devi brought in and Hartley agreed to open. The admissions dataset does not appear at all, because Sam may not read it.

Sam

The console shows why that table is open

Munitas console · Datasets · /datasets

In the console, the Munitas web page for people, Sam searches the datasets list for the lookup. The list shows the dataset that Devi brought in and that the Cardiology department owns.

Screenshot: The console shows why that table is open

The line for diagnosis_codes says 1 of 1 you can read and gives the access level Published, the most open level. Sam may read this dataset without asking anybody.

Sam

The restricted table is closed, with a reason

Python on Sam’s own computer, using DuckDB

The admissions dataset is at the Raw access level, the most restricted one. Nobody may read it without a lease, which is permission from the owning department's custodian to read it for one stated purpose and for a limited time. Sam tries to read it anyway.

Typed by Sam, in Python on Sam’s own computerDuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.Names used here that are not defined in this box: con was set in step 1.

try:
    print(con.sql('SELECT count(*) FROM lake."admissions".v1'))
except Exception as exc:
    print("Refused:", exc)

What came back

Refused: HTTP Error: GetTableInformation endpoint returned response code Forbidden_403 with message "no role reaches class RAW"

The refusal ends with no role reaches class RAW. Sam's role, researcher, reaches the Published level and nothing above it. Munitas gives the reason instead of a bare error.

Sam

The console shows the same closed table

Munitas console · Datasets · /datasets

The same datasets list in the console now shows the admissions dataset, which the Cardiology department owns.

Screenshot: The console shows the same closed table

The line for admissions says 0 of 1 you can read, and the access level is Raw. The dataset has one version, and Sam may read none of it.

Sam

Sam asks the Cardiology custodian for access

Munitas console · A dataset version · /versions/<id>

On the page of the dataset version, Sam writes what the data will be used for, why something less sensitive would not do, and for how long access is needed. The request goes to Hartley, the custodian of the Cardiology department, because Cardiology owns the data.

Screenshot: Sam asks the Cardiology custodian for access

Under Your access are the questions “What will you use it for?” and “Why do you need this rather than something less sensitive?”, a choice “For how long?” and a button Ask for access, with the line Sent under your name, with everything you have written here.

Hartley

Hartley sees the request in the Cardiology queue

Munitas console · The custodian's home page · /

Hartley's home page lists the requests to read Cardiology's data. Only an approver of Cardiology can decide this one, and Sam cannot approve a request made under Sam's own name.

The card reads Sam (Researcher) wants to read admissions v1, followed by Sam's reason and two buttons, Grant access and Refuse. The small text under them says what granting means: a fixed time, this purpose only, and it ends by itself.

Hartley

Hartley grants the request

Munitas console · The custodian's home page · /

Hartley chooses Grant access. The request leaves the queue, and a lease now exists that lets Sam read this one dataset for the stated purpose.

A notice at the bottom right reads Access granted to Sam (Researcher). The box Currently granted now shows 1. The newest entry under What you have decided says Granted, shows when the lease runs out with Still open until, and offers a Revoke link, which ends the lease early.

Sam

With the lease, the restricted table opens

Python on Sam’s own computer, using the Munitas client library and DuckDB

Back in Sam's Python session, the kind of question that was refused a moment ago now works. Storage takes a few seconds to learn of new access, so the first box asks again until it is answered. The second box looks at three rows.

for _ in range(20):                      # storage takes a few seconds to learn of new access
    try:
        print(con.sql('SELECT count(*) AS admissions FROM lake."admissions".v1'))
        break
    except Exception:
        time.sleep(3)

What came back

┌────────────┐
│ admissions │
│   int64    │
├────────────┤
│        400 │
└────────────┘

print(con.sql("""SELECT admission_id, age, diagnosis_code, length_of_stay_days FROM lake."admissions".v1 LIMIT 3"""))

What came back

┌──────────────┬───────┬────────────────┬─────────────────────┐
│ admission_id │  age  │ diagnosis_code │ length_of_stay_days │
│   varchar    │ int64 │    varchar     │        int64        │
├──────────────┼───────┼────────────────┼─────────────────────┤
│ ADM-0001     │    68 │ I47.1          │                   5 │
│ ADM-0002     │    59 │ I42.0          │                   4 │
│ ADM-0003     │    40 │ I48.91         │                   1 │
└──────────────┴───────┴────────────────┴─────────────────────┘

The first answer counts 400 admissions, one for each hospital stay. Real rows come back: the admission number, the age of the patient, the diagnosis code and the days in hospital. There is no name in the table. It was closed because everything that is brought in starts at the safe default, Raw, until the department that owns it decides, whatever the data holds.