DICOM SEG or RTSTRUCT?

Every AI model that outlines something has to save that outline somewhere. If it saves it as a screenshot with the outline burned in, the radiologist sees a picture and nothing else. If it saves it as a proper DICOM object, the viewer can switch it on and off, measure it, and line it up with the source images. (We made the case against burned-in results in Stop burning AI into the image.)

DICOM gives you two real options for this. DICOM Segmentation (SEG) and the Radiation Therapy Structure Set (RTSTRUCT). I read Herman Oosterwijk's article DICOM RTSTRUCT vs DICOM SEG for AI Results when it came out in July, and it is the clearest short comparison I know. This is our take from the side that builds the viewer.

Two different ideas of what a region is

SEG stores voxel masks per segment; RTSTRUCT stores contour points per slice.

The difference is not cosmetic. The two formats disagree about what a region actually is.

SEG thinks in voxels. A SEG object is a multi-frame image that shares the source series' frame of reference, usually on the same grid (it may use a different one). For each segment and each slice there is a mask frame saying, pixel by pixel, "inside" or "outside". It is defined in the Segmentation IOD (PS3.3 A.51). The Segmentation Image Module allows three types: BINARY (1 bit per pixel), FRACTIONAL (8 bits, so a pixel can hold a probability or a partial occupancy, with a declared maximum value meaning 100%), and the newer LABELMAP, where one frame can carry many segments as label numbers.

RTSTRUCT thinks in polygons. An RTSTRUCT holds a list of named Regions of Interest, each made of contours, and each contour is a list of (x, y, z) points in millimetres in the patient coordinate system. The ROI Contour Module defines contour types like POINT, OPEN_PLANAR and CLOSED_PLANAR. It was designed in the 1990s for radiation oncology, where a physicist or oncologist draws the tumour and the organs at risk slice by slice. Herman points out it went into the standard in 1997, roughly ten years before SEG.

So a SEG says "these voxels". An RTSTRUCT says "this boundary, on this slice". An AI model that produces a probability volume naturally produces the first. Converting it to the second means tracing outlines and throwing information away.

The comparison

Read the comparison as a table

	DICOM SEG	DICOM RTSTRUCT
Encoding	Pixel/voxel masks, one multi-frame object in the source frame of reference	Contours as (x, y, z) point lists per slice, referencing the source images
Precision	Exact to the voxel; FRACTIONAL can carry probability or partial volume	Exact to the drawn polygon; no probability, partial voxels are a matter of interpretation
Overlapping regions	Natural in BINARY and FRACTIONAL SEG, where each segment is its own mask; LABELMAP holds one label per voxel, so it cannot overlap	Awkward: overlaps and holes need care in how contours are combined
3D	Volumetric by design	A stack of 2D contours; the 3D shape is reconstructed by the viewer
File size	Grows with matrix size, slice count and number of segments (compression helps a lot)	Usually small, grows with the number of contour points
Where it is used	AI outputs, quantitative imaging, research pipelines	Radiation therapy planning, contouring tools, increasingly as an AI stopgap
Viewer support	Growing, but patchy in installed PACS, especially older releases	Stored by most PACS and read by nearly all RT software

A quick sense of the file size row. One 512 x 512 binary mask frame is 512 x 512 / 8 = 32,768 bytes uncompressed. A whole-body PET/CT with a few hundred slices and several segments adds up, although most frames are mostly empty and compress well. An RTSTRUCT of the same lesions is usually far smaller because it stores only the edges.

File size, worked through: a SEG frame costs bytes per slice, an RTSTRUCT stores only the edges.

Where the two fit together: SEG as the record, RTSTRUCT only as a derived copy.

Why anyone ships RTSTRUCT for AI at all

Herman's answer is blunt and I agree with it: installed base. Plenty of PACS and workstations store RTSTRUCT and can draw it, because radiation oncology has needed that for decades. SEG display support has been slower, and even where a vendor supports it in the newest release, the hospital may be two upgrades behind. In India, a centre with an older PACS is not going to upgrade because an AI vendor asks nicely.

He also warns implementers that SEG and RTSTRUCT are so different that building one does not get you halfway to the other. From the viewer side that matches: a mask is a texture overlay on the image grid, while contours are a polygon problem full of edge cases with overlaps and holes.

What we look for, as a viewer team

When an AI vendor tells a centre "we output segmentations", these are the questions we would ask, in this order.

What we ask a vendor who says they output segmentations.

1. Which format, and which SEG type? BINARY and FRACTIONAL are the common ones. LABELMAP is newer, and viewer support for it is thinner still.
2. Does it reference the right Frame of Reference and source series? A mask that does not align with the images is worse than no mask.
3. Are the segments labelled with codes, not only free text? "Segment 1" helps nobody.
4. Are the algorithm name and version in the object? The IHE AI Results (AIR) profile asks the display to show this for each result, and the radiologist should be able to see which model drew the line (see our AI results checklist).
5. If it is RTSTRUCT, is there a plan to move to SEG? If the vendor has no SEG roadmap, the centre is locking in the stopgap.

The five questions, in the order we ask them.

The PET lesion case

This is where the question stops being theoretical for us. In our browser viewer we do PET/CT fusion and 3D lesion segmentation using a threshold (the workflow is described in Reading PET/CT in the browser). You set a threshold, and the viewer marks out the lesion in 3D from the voxels above it.

By construction, that result is a set of voxels. It is a mask. There is no polygon anywhere in the maths. So when we think about how a lesion like that should be saved and shared with another system, SEG is the honest representation. It keeps the exact voxels that went into the volume, and a second reader opening it later sees the same lesion, not a traced approximation of it.

Could you convert it to RTSTRUCT for an older PACS? Yes, by tracing the mask outline on every slice. But a small lesion a few voxels across becomes a jagged polygon, and if someone recomputes volume from the contours they may get a slightly different number, depending on how the contours were traced and the volume calculated, from the one the radiologist saw. For follow-up PET, where the change in a lesion matters, that drift is exactly what you do not want.

So my practical opinion is simple. Produce SEG as the source of truth. If a site truly cannot display SEG, generate RTSTRUCT from it as a compatibility copy, and label it clearly as derived. Never make the RTSTRUCT the master.

What this means for a centre

If you are buying AI that draws outlines, ask the vendor for a sample SEG and a sample RTSTRUCT, load both into your current PACS, and see what actually happens. If your PACS shows the RTSTRUCT but not the SEG, you have learnt something useful about your next PACS conversation.

Test both formats on your own PACS before you commit.

So where does that leave a centre or a vendor choosing between the two? SEG is the better representation of what an AI model actually produces, and it is the format the standards work, including IHE AIR, points towards. RTSTRUCT is the format that more installed systems can already display, which is a real advantage today; its disadvantage is that it loses probability, handles overlaps awkwardly and drifts on small lesions. That is why, until SEG display support catches up in the installed base, I suggest treating SEG as the record and RTSTRUCT as a convenience copy, and asking every vendor which of the two they treat as the master.

Ask every vendor which of the two they treat as the master.

Related reading

• DICOM SEG and PET/CT lesion segmentation — the SEG object in more depth, with the PET/CT case.
• Reading PET/CT in the browser — fusion and threshold segmentation in a zero-footprint viewer.
• DICOM SR: AI findings a report can read — where measurements and findings belong, as opposed to masks.
• The AI results checklist we hold our viewer to — what IHE AIR asks a viewer to do with SEG, SR and the rest.

FAQ

What is the main difference between DICOM SEG and RTSTRUCT?

SEG stores regions as voxel masks in the images' frame of reference, so it is exact and volumetric. RTSTRUCT stores regions as contours, lists of 3D points slice by slice, because it was built for radiation therapy planning.

Can AI results be saved as RTSTRUCT?

Yes, and several vendors do it because more PACS can display RTSTRUCT today. It works as a transition, but you lose probability information.

Which is bigger, a SEG file or an RTSTRUCT file?

A SEG is usually larger because it holds a mask frame per slice per segment, though compression shrinks the empty frames a lot. An RTSTRUCT only stores boundary points and is usually small.

Which format suits PET lesion segmentation?

SEG. A threshold-based PET segmentation is a set of voxels, and SEG keeps those voxels exactly, which matters when you compare lesion volume across follow-up scans.