MunitasFeature tourAll walkthroughs

All walkthroughs / Health organisation

Health organisation · Custom pipelines

Register a custom pipeline, and run it

A data engineer in the Cardiology department registers a pipeline of their own that checks cardiology notes, asks the department's custodian for access to a note, and runs the pipeline. It ends in the same gate decision as every other pipeline, which only a reviewer may clear.

18Steps
3People
about 14Minutes to read
Who is involved
Devi
Data engineer in the Health organisation
Hartley
Data custodian for the Cardiology department
Imani
De-identification reviewer in the Health organisation

How to read this page

Words used on this page

Organisation
A company or hospital group that uses Munitas. Each organisation sees only its own data.
Department
A team inside an organisation that owns some of its data, such as Cardiology.
Dataset
A named collection of records, such as a table of patient admissions.
Version
One sealed state of a dataset. Changing the contents means making a new version, never editing an old one.
Sealed
Finished and closed for good, so that it can never be edited.
Access level
How restricted a dataset is. Raw is the most restricted, and nobody may read it without permission. Published is the most open.
Custodian
A person who holds the data custodian role, which allows deciding who may read data. A custodian decides for the departments they are an approver for.
Request
A written ask to read one dataset for one stated purpose. The custodian of the department that owns the dataset decides it.
Purpose
The reason a person states for reading data. A lease covers that purpose only, unless the custodian chooses otherwise.
Lease
Permission, given by the custodian, to read one dataset for one stated purpose and for a limited time.
Pipeline
A series of steps that Munitas runs on a dataset version, each step after the ones it depends on. The last step is a gate decision.
Run
One execution of an agent or a pipeline, started for a stated purpose.
Gate decision
The final step of a pipeline, where a person decides whether a dataset version may be released to a more open access level. The person who started the run may not decide it.
De-identification reviewer
A role that allows a person to review data from which identifying details have been removed, and to decide whether it may be released.
Open for annotation
An access level above Under review, where people who label data may work with it.
Release
A decision to let a dataset version be read more widely than before. A release changes who may read it and moves no data.
Under review
The access level of something that no person has reviewed. Nobody may use it until a reviewer decides that it may be released.
PART ONE

Devi registers a pipeline of their own

A pipeline is a series of steps that Munitas runs on a dataset version, each step after the ones it depends on. Munitas offers a starter pipeline, De-identify these recordings, but an organisation may need checks that no starter can know about. Devi, a data engineer in the Cardiology department, needs cardiology notes checked for the fields that Cardiology's quality reporting requires, so Devi registers a pipeline for that.

1
Devi
Devi opens the Pipelines page
Munitas console · Pipelines · /pipelines

The page lists the pipelines registered in the Health organisation, each with an owning department and a version number. Nobody in Health has registered one yet, so the list is empty and the page says so.

Screenshot: Devi opens the Pipelines page
The page says “No pipelines.” and, under it, Nobody has registered one in this organisation yet. The button Register a pipeline is on the page, and the sentence under the heading says that each version is sealed on upload and cannot be changed afterwards.
2
Devi
Devi names the pipeline and its department
Munitas console · Pipelines · Register a pipeline

Devi types a name and chooses Cardiology as the owner. A department is a team inside an organisation that owns some of its data. Ownership says which department is answerable for what the pipeline does. The name ends in a date and a number so that each example has a name of its own.

Screenshot: Devi names the pipeline and its department
Owned by reads Cardiology. The line under the heading reads Names an owner. No version exists yet, so nothing can run until a pipeline definition and its scripts are uploaded on the pipeline’s own page. A pipeline definition lists the steps, and says which steps each one depends on.
3
Devi
The pipeline exists, but nothing can run yet
Munitas console · Pipelines · the pipeline's own page

Registering created a name and an owner and nothing else. The page says so, and offers the upload of the first version below.

Screenshot: The pipeline exists, but nothing can run yet
Owned by Cardiology. Registered by Devi sits under the name. The panel reads No versions, and under it Upload a config and its scripts below to seal the first one.
4
Devi
Devi chooses the two files of a version
Munitas console · Pipelines · Upload the next version

A version is two files. The first, cardiac-note-qc.yaml, names each step in order and says which steps each one depends on. The second is a zip of the scripts that those steps name. This pipeline has two steps: check_notes, Devi's own script, and decide, a gate that comes last.

Screenshot: Devi chooses the two files of a version
Pipeline config (.yaml) shows cardiac-note-qc.yaml, Scripts (.zip) shows a file ending in -scripts.zip, and Upload version is now active.
5
Devi
Version 1 is finished
Munitas console · Pipelines · the pipeline's own page

Munitas checked that every step's dependencies and every script that a step names were present, and then sealed version 1, which means finished and closed for good. A sealed version is never edited: a change is registered as the next version.

Screenshot: Version 1 is finished
v1 and a short code identify the version. Under it, check_notes (script: steps/check_notes.py) and decide (gate) waits for check_notes show the two steps and that decide depends on check_notes. The heading below reads Upload the next version, and says there is no editing in place.
PART TWO

A dataset comes in, and Devi has to ask

Registering a pipeline gives Devi no right to read any data with it. A cardiology note still has to be registered and sealed as a dataset, and a request to read it goes to the custodian of the department that owns it. A custodian is the person a department trusts to decide who may read its data.

6
Devi
Devi registers a dataset for a cardiology note
Munitas console · Bring a dataset in · /datasets/register

A dataset is a named collection of records, and Munitas decides who may read it. Devi names it, gives Cardiology as the owner, and leaves the access level at its default. The form does not ask what kind of data it is, because the platform reads that from the files once they are sealed. The access level says how restricted a dataset is, and Raw is the most restricted level.

Screenshot: Devi registers a dataset for a cardiology note
Owned by reads Cardiology, the origin reads Collected by this organisation, regulated, and the access level reads Raw (the safe default). The small text beside Register says that nothing is readable yet.
7
Devi
Devi uploads a note and seals version 1
Munitas console · Bring a dataset in · upload and seal

Devi uploads consult-note.txt, a one-line cardiology note, and seals the version. After sealing, nothing can be added to it. A later upload becomes a new version.

Screenshot: Devi uploads a note and seals version 1
Uploaded sits next to consult-note.txt, and a green panel reads Sealed as version 1, 1 file. Nothing about it can change from here; a later upload becomes a new version. Below it is a link, Run a pipeline against this version.
8
Devi
Devi opens the dataset and has to ask
Munitas console · Version detail · /versions/<id>

The version page offers a place to run a pipeline, and under Your access it explains that the data is more sensitive than Devi's own role may read. A request is a written ask to read one dataset for one stated purpose. Devi fills in the purpose, the reason that something less sensitive would not do, and how long access is needed.

Screenshot: Devi opens the dataset and has to ask
Your access reads This is more sensitive than your role reads, so Hartley decides. The purpose reads run cardiac-note-qc against this version to check note quality before wider release, and the button reads Ask for access.
9
Devi
The request goes to the custodian of Cardiology
Munitas console · Version detail · after sending the request

Devi cannot approve the request, whatever the request says. The page reports that the request has gone to the person answerable for this data, and that nothing is readable until that person grants it.

Screenshot: The request goes to the custodian of Cardiology
Your access now reads Your request has gone to the person answerable for this data. Nothing is readable until they grant it, and the form has gone.
PART THREE

The custodian of Cardiology decides

Hartley is the data custodian of the Cardiology department, the person answerable for who may read what that department owns, including a note that a Cardiology engineer registered.

10
Hartley
Hartley signs in and finds Devi's request waiting
Munitas console · Home · the custodian's home page

Hartley's home page lists the requests to read Cardiology's data. The first card carries everything Devi wrote.

Screenshot: Hartley signs in and finds Devi's request waiting
Waiting for your decision counts 1. The card reads Devi (Data engineer) wants to read cardiology-intake-notes v1 with the label Raw, Devi's reason, the purpose, and for 24 hours. Beside the two buttons, the text reads Granting gives Devi 24 hours, for this purpose only, and it ends by itself.
11
Hartley
Hartley grants access
Munitas console · Home · after granting

Granting creates a lease, which is permission to read one dataset for one stated purpose and for a limited time. The request leaves the waiting list and moves into the history of decisions.

Screenshot: Hartley grants access
A notice at the bottom right reads Access granted to Devi (Data engineer). Waiting for your decision counts 0 and Currently granted counts 1. The top decision reads Granted, with a time until which it stays open and a Revoke link.
PART FOUR

Devi runs the pipeline

With the lease in place, the version page tells Devi what the access covers, and Devi can choose the pipeline that Devi registered.

12
Devi
The page now shows the access that Devi has
Munitas console · Version detail · /versions/<id>

In place of the request form, Your access states what the lease covers and until when.

Screenshot: The page now shows the access that Devi has
Your access reads You can read this until a date for run cardiac-note-qc against this version to check note quality before wider release. The request form has gone.
13
Devi
Devi chooses the new pipeline from the list
Munitas console · Version detail · Run a pipeline

The Pipeline list used to offer only the starter pipeline. It now also offers the pipeline that Devi registered in Part One.

Screenshot: Devi chooses the new pipeline from the list
The list reads cardiac-note-qc followed by the date and number, and (v1), which is the version sealed in step 5. The Start button is next to it.
14
Devi
Devi starts the run
Munitas console · Pipeline run · /pipeline-runs/<id>

A run is one execution of a pipeline. The run page says what is running, who started it, and what has and has not happened. Nothing is decided or released at this point.

Screenshot: Devi starts the run
The heading reads Running cardiac-note-qc against the dataset, and the line under it reads Running now. This page updates itself. Started by reads Devi. What happens next reads Nothing has been decided, and nothing has been released.
15
Devi
Both steps finish, and the run points to a decision
Munitas console · Pipeline run · /pipeline-runs/<id>

The list of steps is read from what the run reported, so the console shows the two steps of Devi's pipeline and not a fixed list. Under it, the page says that a reviewer has to decide whether the note may be released, and that the reviewer cannot be whoever started the run.

Screenshot: Both steps finish, and the run points to a decision
Steps lists check_notes and decide, each marked Done. The line under the heading reads Finished. Nothing has been released yet: a reviewer decides that. The measurement reads pass: cardiac note fields present and well-formed, and a link reads Open the decision.
PART FIVE

Every pipeline ends with a person's decision

The last step of a pipeline is a gate decision: a person decides whether a dataset version may be released to a more open access level. It works the same way for a pipeline that Devi wrote as for the starter pipeline, and only a de-identification reviewer may clear it. A de-identification reviewer is a role that allows a person to review the result of a pipeline and release it or hold it back.

16
Devi
Devi cannot approve the result of Devi's own run
Munitas console · Gate decision · /gates/<id>

The decision would move the note from Raw to Open for annotation, an access level above Under review where people who label data may work with it, and the page says this cannot be undone. Devi started the run, so Devi may not clear it.

Screenshot: Devi cannot approve the result of Devi's own run
The state reads Waiting for a decision. The text above the buttons reads “You started this run, so somebody else has to decide it.” and Release to open for annotation and Hold it back are both pale and cannot be pressed.
17
Imani
Imani, the reviewer, writes down why
Munitas console · Gate decision · /gates/<id>

Imani holds the de-identification reviewer role. A reason has to be written before either button can be pressed, so that whoever reads the record later has the reasoning and not only the outcome.

Screenshot: Imani, the reviewer, writes down why
The panel at the bottom left of the menu reads Imani and, under the name, De-identification reviewer. The reason reads Required registry fields are present and well-formed; safe to open for annotation, and both buttons are now active.
18
Imani
Imani releases the note
Munitas console · Gate decision · after releasing

The note moves from Raw to Open for annotation. The decision and its reason are now part of the record on this page.

Screenshot: Imani releases the note
The state reads Released. The decision reads Released by Imani followed by the reason, and a notice at the bottom right reads Released to open for annotation.