How to read this page
- Every step names the tool or the part of Munitas where it happens. The label under the title says where, and the step reads in the same order each time: what is happening, the evidence, and then what to look for in it.
- Console steps happen in the Munitas console, the web app that people open in a browser. They show real screenshots, and clicking one enlarges it.
- Python steps happen in Python, on the person’s own computer, in a notebook or in a terminal. Python is a window where a person types a line and sees the answer straight away. From there the person uses the Munitas client library, which sends requests to Munitas, and DuckDB, a free tool for analysing tables. Each dark box on the page is one of these steps.
- In a dark box, the top part is exactly what was typed and the bottom part is exactly what came back.
meis the client library signed in as the person named on the label, andcustodianis the client library signed in as the custodian. - Every step was run for real by a script that types each one, so the answers are genuine and nothing has been edited. Every name of a person, patient or card holder in the example data is invented.
Words used on this page
- Organisation
- A company or hospital group that uses Munitas. Each organisation sees only its own data.
- Department
- A team inside an organisation that owns some of its data, such as Cardiology.
- Dataset
- A named collection of records, such as a table of patient admissions.
- Custodian
- A person who holds the data custodian role, which allows deciding who may read data. A custodian decides for the departments they are an approver for.
- Access level
- How restricted a dataset is. Raw is the most restricted, and nobody may read it without permission. Published is the most open.
- Lease
- Permission, given by the custodian, to read one dataset for one stated purpose and for a limited time.
- Query
- A question about a dataset, written in a language called SQL.
- Sensitivity label
- How identifying a column is. none does not identify anybody, quasi is a detail that could help identify a person when combined with others, and phi is protected health information.
- Token
- A short-lived password that also states why the person wants to read.
- Sealed
- Finished and closed for good, so that it can never be edited.
- Iceberg table
- An open way of storing a table that tools such as DuckDB can read directly.
- Claim
- A statement, made by the person who brings a dataset in, about how sensitive it is. A different approver of the owning department has to confirm it.
- Version
- One sealed state of a dataset. Changing the contents means making a new version, never editing an old one.
- Container
- An isolated environment, with no network access of its own, where Munitas runs an agent.
- Filter
- Keeping only the rows of a table that match values the custodian names, such as one patient’s id, so that other people’s rows do not leave with them.
A closed table, and a request for access
The Fraud Operations department of a finance organisation owns a dataset of card transactions. Every record names the account holder and the last four digits of the card, so the dataset is closed to most people. An analyst wants to review it. This part shows the analyst being stopped, asking for access, and being answered by a different person.
Omar is an analyst in the Finance organisation. To start, Omar asks the platform for a token, which is a short-lived password that also says why Omar wants to read, here a cross-border fraud review. Omar then connects DuckDB, a free tool for analysing tables, to the platform's catalog, which is the list of tables the platform offers, and asks what is in it. DuckDB runs inside Omar's own Python session and reads the tables from the platform.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.token = me.catalog_token(purpose="cross-border fraud review") # shown once, never printed
import duckdb
con = duckdb.connect()
con.execute("INSTALL iceberg; LOAD iceberg;")
con.execute(f"ATTACH 'finance' AS lake (TYPE ICEBERG, ENDPOINT 'http://localhost:8000/iceberg', TOKEN '{token}')")
print(con.sql("SELECT schema AS dataset, name AS version FROM (SHOW ALL TABLES) ORDER BY 1"))What came back┌───────────┬─────────┐ │ dataset │ version │ │ varchar │ varchar │ ├───────────┼─────────┤ │ merchants │ v1 │ └───────────┴─────────┘
merchants, a public list of merchants and their categories. The transactions dataset does not appear at all, because Omar may not read it.The console is the platform's web page for people. Signed in as Omar, the datasets list shows merchants, the public list of merchants, which the Risk and Compliance department owns.
The transactions dataset is at the Raw access level, the most restricted one. Nobody may read it without a lease, which is permission from the owning department's custodian to read it for one stated purpose and for a limited time. Omar tries to read it anyway.
con was set in step 1.try:
print(con.sql('SELECT count(*) FROM lake."transactions".v1'))
except Exception as exc:
print("Refused:", exc)What came backRefused: HTTP Error: GetTableInformation endpoint returned response code Forbidden_403 with message "no role reaches class RAW"
no role reaches class RAW. Omar's role, analyst, reaches the Published level and nothing above it. The platform gives the reason instead of a bare error. The line also names one lease that was revoked earlier, because this example was recorded more than once on the same data and the platform remembers Omar's earlier lease. Only the most recent revoked lease is named, however many there have been.The same datasets list in the console now shows the transactions dataset, which the Fraud Operations department owns and which holds the card transactions. The list also holds high-value-foreign-transactions, a dataset made from these transactions in an earlier recording of this walkthrough.
On the dataset's page, Omar writes what the data will be used for, why something less sensitive would not do, and for how long access is needed. The request goes to Marcus, the custodian of the Fraud Operations department, because Fraud Operations owns the data.
Marcus's home page lists the requests to read Fraud Operations' data. Only an approver of Fraud Operations can decide this one, and Omar cannot approve a request made under Omar's own name.
card-transaction-log, which this walkthrough does not touch.Marcus chooses Grant access. The request leaves the queue, and a lease now exists that lets Omar read this one dataset for the stated purpose.
Back in Omar's Python session, the kind of question that was refused a moment ago now works. Every name in the table is invented for this example.
con was set in step 1.print(con.sql("""SELECT txn_id, account_holder, card_last4, amount, country FROM lake."transactions".v1 LIMIT 3"""))What came back┌─────────┬────────────────┬────────────┬────────┬─────────┐ │ txn_id │ account_holder │ card_last4 │ amount │ country │ │ varchar │ varchar │ varchar │ double │ varchar │ ├─────────┼────────────────┼────────────┼────────┼─────────┤ │ T-00001 │ Chidi Berg │ 6344 │ 182.4 │ GB │ │ T-00002 │ Noor Dubois │ 2082 │ 24.01 │ GB │ │ T-00003 │ Elif Fontaine │ 4979 │ 92.46 │ GB │ └─────────┴────────────────┴────────────┴────────┴─────────┘
account_holder and card_last4, which identify a person and a card. That is why the dataset is restricted, and why anything made from it has to stay just as careful.Making a new dataset from a query
Omar only needs the large payments made outside the home country. Munitas lets a person make a new dataset by writing a query over datasets the person may already read. The platform runs the query itself, not the person's computer, and holds the new dataset to the same rules as the data it came from.
Omar writes a query that joins the restricted transactions with the public list of merchants and keeps payments over 300 made outside the United States, then asks the platform to plan it. Nothing runs yet. The platform replies with the columns the new dataset would have, and the sensitivity label each one must carry: none for a column that does not identify anybody, and quasi for a detail that could help identify a person when combined with others.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar.sql = """SELECT t.txn_id, t.amount, t.country, t.occurred_at, m.category, m.high_risk, t.flagged
FROM t JOIN m ON m.merchant_id = t.merchant_id
WHERE t.amount > 300 AND t.country <> 'US'"""
draft = me.draft(inputs=[{"dataset": "transactions", "alias": "t"}, {"dataset": "merchants", "alias": "m"}], sql=sql, name="large-foreign-payments-1011-0019",
primary_key=["txn_id"], purpose="cross-border fraud review")
for f in draft["fields"]:
print(f"{f['name']:<22}{f['type']:<8}{f['sensitivity']}")
print("The new dataset would be class", draft["output_class"])What came backtxn_id string none amount float none country string quasi occurred_at string quasi category string none high_risk bool none flagged bool quasi The new dataset would be class RAW
category and high_risk, which come from the public list of merchants, are none. country, occurred_at and flagged, which come from the transactions, are quasi. The last line says the new dataset would be RAW, because it takes the strictest level among its inputs.A label can be raised but not lowered. Lowering one would claim that the data is safer than the data it came from, and that claim needs somebody other than the person who wrote the query. Omar tries to mark the country column as none anyway.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar.Names used here that are not defined in this box: MunitasError is the error the client library raises when Munitas refuses a request.probe = me.draft(inputs=[{"dataset": "transactions", "alias": "t"}], sql="SELECT txn_id, country FROM t", name="large-foreign-payments-1011-0019-probe",
primary_key=["txn_id"], purpose="cross-border fraud review")
try:
me.confirm(probe, sensitivities={"country": "none"})
except MunitasError as exc:
print("Refused:", exc)What came backRefused: HTTP 422: 'country' is computed from fields marked 'quasi', so it cannot be marked 'none'. A sensitivity can only be raised here: lowering one needs the agreement of somebody other than the person who wrote the query
quasi, so it cannot be marked none.After Omar confirms, the platform copies the input files into a sandbox, an isolated container with no network connection and no passwords. The query runs there, and the platform checks every row of the result against the plan. The query never runs on Omar's computer. Omar's session only waits for the answer.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar. To run the query, Munitas starts it in a sandbox, an isolated container with no network, and writes the result back as a new dataset.Names used here that are not defined in this box: draft was set in step 9.confirmed = me.confirm(draft)
result = me.wait(confirmed["id"])
print("Status:", result["status"])
print("New dataset:", result["target_name"], "|", result["output_class"])What came backStatus: succeeded New dataset: large-foreign-payments-1011-0019 | RAW
The new dataset belongs to the same department as the data it came from, so Marcus is still the person who decides who may read it. Omar made it, so Omar can read it straight away without asking anybody.
The platform sealed the result, which means it is finished and closed for good, so it can never be edited. It is also an Iceberg table, an open way of storing a table that tools such as DuckDB read directly. Omar queries it.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.Names used here that are not defined in this box: token was set in step 1.con = me.attach_duckdb(token)
print(con.sql("""SELECT txn_id, amount, country, category, flagged FROM lake."large-foreign-payments-1011-0019".v1 ORDER BY amount DESC LIMIT 5"""))
print(con.sql("""SELECT count(*) AS transactions, round(sum(amount), 2) AS total_amount, count(*) FILTER (WHERE flagged) AS flagged FROM lake."large-foreign-payments-1011-0019".v1"""))What came back┌─────────┬─────────┬─────────┬─────────────┬─────────┐ │ txn_id │ amount │ country │ category │ flagged │ │ varchar │ double │ varchar │ varchar │ boolean │ ├─────────┼─────────┼─────────┼─────────────┼─────────┤ │ T-00424 │ 1060.54 │ GB │ dining │ false │ │ T-00170 │ 801.76 │ BR │ electronics │ false │ │ T-00388 │ 735.71 │ NG │ dining │ false │ │ T-00580 │ 705.47 │ DE │ travel │ false │ │ T-00180 │ 675.66 │ DE │ jewellery │ false │ └─────────┴─────────┴─────────┴─────────────┴─────────┘ ┌──────────────┬──────────────┬─────────┐ │ transactions │ total_amount │ flagged │ │ int64 │ double │ int64 │ ├──────────────┼──────────────┼─────────┤ │ 31 │ 15103.46 │ 0 │ └──────────────┴──────────────┴─────────┘
Every version of a dataset records what produced it: the step, who ran it, and the exact versions of the datasets it was made from.
Anybody auditing the new dataset can follow it back. The platform keeps the purpose, the datasets and versions it was made from, and the text of the query.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar.Names used here that are not defined in this box: result was set in step 11.record = me.derivation(result["id"])
print("Purpose :", record["purpose"])
print("Inputs :", ", ".join(f"{i['dataset']} version {i['version']} ({i['class']})" for i in record["inputs"]))
print("Query :")
print(record["sql"])What came backPurpose : cross-border fraud review Inputs : transactions version 1 (RAW), merchants version 1 (PUBLISHED) Query : SELECT t.txn_id, t.amount, t.country, t.occurred_at, m.category, m.high_risk, t.flagged FROM t JOIN m ON m.merchant_id = t.merchant_id WHERE t.amount > 300 AND t.country <> 'US'
The rules still apply to the new dataset
A new dataset is only safe if nobody can use it to get around the rules. This part tests that with a colleague, with a query that reaches too far, and with the custodian taking the access back.
Lena is a data engineer in the same organisation, with no lease on the new dataset. The new dataset is as restricted as the data it came from, so Lena cannot open it.
colleague is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Lena. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.colleague_token = colleague.catalog_token(purpose="curiosity")
other = colleague.attach_duckdb(colleague_token)
try:
print(other.sql('SELECT count(*) FROM lake."large-foreign-payments-1011-0019".v1'))
except Exception as exc:
print("Refused:", exc)What came backRefused: HTTP Error: GetTableInformation endpoint returned response code Forbidden_403 with message "no role reaches class RAW"
no role reaches class RAW. Working in the same organisation is not enough.A query may read the datasets it names and nothing else. Omar tries one that reads a file from the machine instead.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar.Names used here that are not defined in this box: MunitasError is the error the client library raises when Munitas refuses a request.try:
me.draft(inputs=[{"dataset": "transactions", "alias": "t"}], sql="SELECT content AS txn_id FROM read_text('/etc/passwd')",
name="large-foreign-payments-1011-0019-escape", primary_key=["txn_id"], purpose="cross-border fraud review")
except MunitasError as exc:
print("Refused:", exc)What came backRefused: HTTP 422: that query reaches outside the datasets it declared (a file or an address). A query may read only its declared inputs
The decision log lists every request to read data. Each request is recorded twice: once for whether it was allowed, and once for whether access was actually given. Here the log is filtered to the requests that were refused.
Marcus ends the lease. Omar immediately loses the transactions dataset, and the new dataset too, because Omar's access to the new dataset rested on the same lease.
custodian is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Marcus.Names used here that are not defined in this box: lease is the lease the custodian approved earlier in the same session.custodian.revoke(lease)
print("The lease is withdrawn.")What came backThe lease is withdrawn.
me is the Munitas client library, a small Python package that asks Munitas for things on the person’s behalf, signed in as Omar. DuckDB runs in this Python session. It asks Munitas for the table. If Munitas allows it, DuckDB receives a short-lived key and reads the table directly.Names used here that are not defined in this box: token was set in step 1; time is Python’s standard time module.time.sleep(3)
con = me.attach_duckdb(token)
for name in ["transactions", "large-foreign-payments-1011-0019"]:
try:
print(con.sql(f'SELECT count(*) FROM lake."{name}".v1'))
except Exception as exc:
print(f"{name}: refused, {str(exc).split('message ')[-1]}")What came backtransactions: refused, "lease bcc8bfb3-97b5-47ac-891d-fc4e58e4dcf9 revoked; no role reaches class RAW" large-foreign-payments-1011-0019: refused, "lease fe97cb6c-7169-44ca-bf7f-cd679eae84fb revoked; no role reaches class RAW"
The new dataset is still listed, so Omar can still know that it exists, but Omar can no longer read it.
