MunitasFeature tourAll walkthroughs

All walkthroughs / Health organisation

Health organisation · Custom pipelines

Test a new pipeline version on a development sample

A data engineer has written a pipeline version that is still in development, so it may run only on development data. The engineer asks for a sample of the locked recordings, a custodian who did not ask approves fewer, and the platform draws them without anybody choosing. The pipeline version then runs for real on the sample and is refused on the real recordings. The sample ends, and recordings that hold no real person are declared synthetic.

15Steps
2People
about 12Minutes to read
Who is involved
Devi
Data engineer in the Health organisation
Hartley
Data custodian for the Cardiology department

How to read this page

Words used on this page

Organisation
A company or hospital group that uses Munitas. Each organisation sees only its own data.
Department
A team inside an organisation that owns some of its data, such as Cardiology.
Dataset
A named collection of records, such as a table of patient admissions.
Custodian
A person who holds the data custodian role, which allows deciding who may read data. A custodian decides for the departments they are an approver for.
Department approver
A custodian who is listed for one department and may approve access to its data and confirm claims about it. A department can have several, and any one of them can act on their own.
Sealed
Finished and closed for good, so that it can never be edited.
Version
One sealed state of a dataset. Changing the contents means making a new version, never editing an old one.
Pipeline
A series of steps that Munitas runs on a dataset version, each step after the ones it depends on. The last step is a gate decision.
Run
One execution of an agent or a pipeline, started for a stated purpose.
Custom step
A step of a pipeline whose script was written by the person who registered the pipeline, instead of one of the steps Munitas provides. It runs in a closed container with no internet, no key and no storage address, and can only ask the platform for the recordings it is allowed to read and send back the files it makes.
Stage
Where a version of a pipeline is in its life. In development means the version runs only on development data, which is a small sample or recordings with no real person in them. In production means it runs on any data the department allows. Retired means it has ended for good and cannot be run again.
Development sample
A few recordings drawn by the platform from a sealed version of locked recordings, for building and testing a pipeline version that is still in development. It names the same locked recordings as the original and copies none, and it ends on its own.
Locked recording
A recording stored scrambled under a key made for the one patient it is about. Reading it back needs the platform to open it, and destroying the patient’s key later makes it unreadable for good.
Synthetic
Made up, with no real person in it, such as sentences read aloud by volunteers from invented scripts.
Lease
Permission, given by the custodian, to read one dataset for one stated purpose and for a limited time.
Container
An isolated environment, with no network access of its own, where Munitas runs an agent.
PART ONE

Devi asks for a sample of the recordings

A pipeline is a series of steps that Munitas runs on a version of a dataset. A pipeline version that has a custom step, a step whose script was written by the person who registered the pipeline, starts in development, which means it may run only on development data. For recordings of patients, development data is a small sample of them, or recordings that hold no real person. This part shows where a sample comes from.

1
Devi
Devi asks for a development sample of the recordings
Munitas console · Version detail · /versions/<id>

Devi is a data engineer in the Cardiology department of the Health organisation. A department is the team inside an organisation that owns some of its data. Devi has written a pipeline version whose step makes the first half second of each recording silent, and it is in development. Devi opens the sealed version of the consultation recordings, which means the version that is finished and closed for good, so that it can never be edited. Under the heading Development samples, Devi says how many recordings are wanted, for how many days, and what the sample is for. Devi cannot choose which recordings: the platform draws them.

Screenshot: Devi asks for a development sample of the recordings
The heading Development samples explains that a sample is a few of these files, drawn by the platform, that names the same locked files and copies none. Under Ask for a sample, “Draw a sample for me.” is selected, the boxes read 4 files and 5 days, and the purpose reads Building a step that makes the first half second of each recording silent. The limits are written beside the boxes: at most 50 files and at most 14 days.
2
Devi
The request waits for an approver
Munitas console · Version detail · /versions/<id>

Pressing the button records the request and does nothing else. A sample needs the agreement of an approver of the department that owns the recordings, and the approver cannot be the person asking. An approver is a data custodian, a person who may decide who reads a department’s data, who is listed for that department. Devi may withdraw the request while it waits.

Screenshot: The request waits for an approver
Under Your samples a row reads Waiting for an approver and Asked for 4 files for 5 days, to build and test with, and offers the button Withdraw this request. The form above now offers only “Name the files that went wrong.”, because a request to draw a sample is already waiting.
3
Hartley
Hartley sees the request, and cannot choose the recordings
Munitas console · Version detail · /versions/<id>

Hartley is the data custodian for Cardiology, so the same page shows the request to Hartley, under Waiting for your decision. It says who asked, how many recordings, for how many days and why. Hartley may lower the number or the days, and may not raise them. Hartley may not pick recordings either: the platform draws them, so nobody can steer a sample towards particular patients.

Screenshot: Hartley sees the request, and cannot choose the recordings
Under Waiting for your decision a yellow box reads “Devi asks for 4 files for 5 days, to build and test with.”, then the purpose. Two boxes, Files and Days, each say that you may lower this, and a third reads A note, recorded with your decision. The text says You cannot choose the files: the platform draws them. The buttons are Approve and Decline. A white box above them lets an approver declare the files synthetic, which Part Four uses.
4
Hartley
Hartley approves fewer recordings, and the platform draws them
Munitas console · Version detail · /versions/<id>

Hartley lowers the request to 3 recordings for 3 days, adds a note, and approves. The platform then draws the recordings by one fixed method: at most one recording from each patient, spread across short, medium and long recordings, and picked by a value that comes out the same every time for the same person asking for the same purpose. Nobody chooses, including Hartley. The platform seals a new version, the sample, that names the same locked recordings as the original. No audio is copied.

Screenshot: Hartley approves fewer recordings, and the platform draws them
Under In force for these files one line reads Devi: 3 files (1 short, 1 medium and 1 long), ends followed by a date three days away, and offers End this sample. The line shows how many recordings came from each length and names nobody but Devi: no recording and no patient.
PART TWO

Devi uses the sample

Devi now runs the pipeline version on the sample. Running it on the real recordings is still refused, which shows what development means.

5
Devi
Devi opens the sample, which says what it is
Munitas console · Version detail · /versions/<id>

The platform made the sample as a version of its own, in a dataset named after the recordings with dev-sample and a short code added. A dataset is a registered, owned collection of data. Devi opens the sample from the list of Devi’s own samples. The page counts 3 recordings about 3 people and says that the sample is for the person it was drawn for and for nobody else. Lower down, the section Where this came from names the platform’s own step that made the sample, and links to the recordings it was made from.

Screenshot: Devi opens the sample, which says what it is
Under Locked files the counts read Files 3 and People 3. Under Development sample a green box reads “This is a development sample of 3 files, drawn for you.”, then the date it ends, and says that it names the same locked files as the files it was drawn from and copies none of them. Under Where this came from, Step reads development_sample, Code version reads sample:v1 and Environment reads platform.
6
Devi
On the real recordings the same pipeline version is refused
Munitas console · Version detail · /versions/<id>

Devi opens the real recordings again, chooses the pipeline version and presses Start. The platform refuses before it starts anything. A pipeline version in development may run only on development data, and the real recordings are not development data for anybody. Nothing is left behind: no run, no file and no record of an attempt to read.

Screenshot: On the real recordings the same pipeline version is refused
A red box reads Nothing was started. Fix these first: and says that this version is in development, so it can be run only on development data: files declared synthetic, or a development sample drawn for you. It names the custom step silence, says that on any other data an approver of the department that owns the pipeline has to check the version and move it to production first, and ends Ask for a sample on the version’s page, under Development samples.
7
Devi
On the sample the pipeline version runs for real
Munitas console · Pipeline run · /pipeline-runs/<id>

Devi opens the sample, chooses the same pipeline version and presses Start, and this time it runs. The script runs in a container, which is an isolated environment. This one has no internet and holds no key and no storage address. It asks the platform for its recordings and is told of exactly the 3 in the sample. The platform finds the stored files that the sample names, unlocks each with its own patient’s key inside the platform, and hands the sound over. What the step writes back becomes a new version that waits for a person’s review.

Screenshot: On the sample the pipeline version runs for real
The heading reads Running, the name of the pipeline, against, and the name of the recordings followed by dev-sample and a short code. The page says Finished. Under Steps, silence reads Done and has a link Open the version it made, and decide also reads Done.
8
Devi
Devi sees the files the step made
Munitas console · Version detail · /versions/<id>

Devi follows the link Open the version it made. A version is one sealed copy of a dataset, here the files the step wrote. The page has a section called Files. It lists the three recordings the step made silent, each marked Locked, which means the platform keeps the file scrambled with the key of the one patient it came from, and unscrambles it only inside the platform. Devi may open them because Devi started the run, and the run read a sample drawn for Devi. A line under the heading says when the files are deleted: after seven days at the latest, or sooner if the sample they came from ends.

Screenshot: Devi sees the files the step made
Under Files the page reads “A run you started made these from development data, so you may open them while that data is in force for you.” and These files are deleted on a date three days ahead. Three rows end in silenced.wav, each with its size, the word Locked and a Play button. No patient name and no key appear anywhere.
9
Devi
Devi plays one of them
Munitas console · Version detail · /versions/<id>

Devi presses Play on the first file. A sound player appears in the page and loads the recording. The platform unscrambles the file as it sends it, so the browser receives ordinary sound and never the key or the place where the file is stored. Each opening is written down: who opened which file, why the platform allowed it, and how many bytes were sent.

Screenshot: Devi plays one of them
Under the first row a sound player shows a pause button, the time 0:00 / 0:01 and a volume control. There is no download button.
10
Hartley
Hartley, who approved the sample, cannot open what Devi made
Munitas console · Version detail · /versions/<id>

Hartley is the data custodian, the approver for the Cardiology department, and approved the sample. Hartley opens the same page. The files are listed, so Hartley can see that they exist and how large they are, but there is no Play button. A sample is for the person it was drawn for, and what a run makes from it is that person’s to open. Hartley could open real recordings only under a lease, which is a time-limited permission asked for on the page of the recordings and decided by a second person.

Screenshot: Hartley, who approved the sample, cannot open what Devi made
A yellow box reads “You cannot open these files.” followed by the reason. The three rows are listed with Locked and no button.
PART THREE

The sample ends

A sample ends when its days are over, or sooner when the person it was drawn for, or an approver, ends it. Nothing is left running on it afterwards.

11
Devi
Devi ends the sample when the work is done
Munitas console · Version detail · /versions/<id>

Devi presses End this sample on the page of the real recordings. The row changes to Ended, and the form for asking appears again with Draw a sample for me selected, because the earlier request is over and a new one may be made.

Screenshot: Devi ends the sample when the work is done
Under Your samples the row reads Ended and 3 files, to build and test with, and has no buttons. The form above offers both kinds of sample again, with “Draw a sample for me.” selected.
12
Devi
A run on an ended sample is refused
Munitas console · Version detail · /versions/<id>

Devi opens the ended sample and tries to run the pipeline version on it. The platform refuses at the start, for the same reason as on the real recordings: the sample is no longer development data for Devi. A step that is already running when a sample ends is refused at its next request.

Screenshot: A run on an ended sample is refused
A red box reads Nothing was started. Fix these first: and gives the same reason as before, that the version is in development and can be run only on development data.
13
Devi
When the sample ends, what was made from it ends too
Munitas console · Version detail · /versions/<id>

Devi opens the version the step made once more. The page now says that the version was made for development and has ended. The platform deletes the files within a minute of the sample ending, so nothing made from a sample outlives it. The version itself stays on record, because a sealed version is never removed, and so does a note of how many files and bytes were freed and why. The real recordings the sample named are not touched: a sample holds only a short list.

Screenshot: When the sample ends, what was made from it ends too
Under Files a single line reads This version was made for development and has ended, with the time, Its files are deleted. The version itself stays on record. No rows and no Play button remain.
PART FOUR

Recordings that hold no real person

Some recordings hold no real person, for example sentences read aloud by volunteers from invented scripts. They need no sample.

14
Hartley
Hartley declares a version of recordings synthetic
Munitas console · Version detail · /versions/<id>

Synthetic means made up, with no real person in it. An approver of the owning department may declare a version of recordings synthetic, and must give a reason. Hartley opens a version of recordings that volunteers read aloud and types why.

Screenshot: Hartley declares a version of recordings synthetic
Under Development samples a white box reads If these files hold no real person, declare them synthetic. Then any pipeline version in development may run on them, and nobody needs a sample. The reason box reads Read aloud by volunteers from invented scripts, so no patient is in them, beside the button Declare these files synthetic.
15
Hartley
The declaration is recorded, with who made it and why
Munitas console · Version detail · /versions/<id>

After Hartley presses the button, the page says who made the declaration, when, and why. From then on a pipeline version in development may run on these recordings for anybody in the organisation, because nobody real is in them. A development sample holds real recordings, so it can never be declared synthetic.

Screenshot: The declaration is recorded, with who made it and why
A green box has three lines. Declared synthetic by Hartley on a date and time. Reason given: Read aloud by volunteers from invented scripts, so no patient is in them. No real person is in these files, so a pipeline version in development may run on them.