DS ADR: don't write .raw files

[Tom-Willemsen]

I believe we have now done enough research with downstream consumers incl. GudRun, Mantid, etc to definitively rule out writing .raw files from the DS system.

[Tom-Willemsen]

@martyngigg fyi as we were discussing this informally the other day.

[ChrisM-S]

The arguments in the proposed decision to drop ".RAW" format files from the file writer output seem sound. There is however a second function which the .RAW file has fulfilled and it is important for the downstream archiving of the instrument data files (produced at the end of a run) that this functionality is maintained.

At the end of each run, the .RAW file is currently augmented with a second windows NTFS file-stream. This is not normally read by applications and so forms a location for validation metadata about the contents of all files produced at the time the experimental run ends. A particular advantage of this is that a validation checksum of the .RAW file can be stored in this file stream without affecting the calculated checksum of the file (thus providing an independent check of file integrity across any subsequent file transfers).

The logical preservation of this would be to generate these checksums to a second file stream on the .NXS file by the file writer/end of run script (whilst the ISIS archive remains on Windows at least). It is possible to write and archive an independent checksum file, but doing this produces a significant risk that the seemingly random file will not be copied and won't follow the data file. It might be good to ensure that if a separate file is created it maintains the ".NXS" extension (e.g. ENGINX00374779_checksum.NXS) this would make it more likely to be copied as a data file and not just ignored, future software could also take note of this and check/validate the binary .NXS data file for integrity in later decades.

A final point is that the "_checksum" file could be linked into an overall NeXus file container and be updated at the end of production of other linked .NXS files. So, at this point the extra Windows file stream mechanism might no longer be needed, though it might still be wise to keep the plain text format of the second file stream for data checks by a non-NeXus application or file system across the suite of .NXS or other files produced on the instrument disk and the time of an "END".

Examples for two instruments

These are different and given as examples, but of note is:

1. that the .RAW file on MUSR no longer has any data content. This can actually be seen in the checksums, both the .RAW and ICPalarm.txt files have the checksum of a zero length file (d41d8cd98f00b204e9800998ecf8427e).
2. On ENGINX, the checksums include intermediate saved files, the .s01 would obviously not be there if the RAW file format is removed, but the .n001 saved NeXus file would (I expect) still be required. Whether it is worth modifying the naming away from an incrementing extension is worth considering. e.g. "ENGINX00374779_001.nxs" is worth considering.

MUSR

 Directory of \\isis\inst$\NDXMUSR\Instrument\data\cycle_26_1

12/06/2026  08:33                 0 MUSR00104406.RAW
               1 File(s)              0 bytes
               0 Dir(s)  6,511,410,020,352 bytes free

C:\Users\cms42>more < \\isis\inst$\NDXMUSR\Instrument\data\cycle_26_1\MUSR00104406.raw:checksum
a15c713eec258ce6271cf91a32c4e26f *MUSR00104406.log
2a33f832509d34a23fd24d67135dac4e *MUSR00104406.nxs
d72763f03585c8175150590d5cb75665 *MUSR00104406.nxs_v1
d41d8cd98f00b204e9800998ecf8427e *MUSR00104406.RAW
d41d8cd98f00b204e9800998ecf8427e *MUSR00104406_ICPalarm.txt
6af8e0ac0e5949184c97941ddaa1f9cd *MUSR00104406_ICPdebug.txt
582a0eb16bb7d4aecffead8ea3318b6b *MUSR00104406_ICPevent.txt
8d682f9377de119f1bbb4c3bd8b786a1 *MUSR00104406_ICPputlog.txt

ENGINX

 Directory of \\isis\inst$\NDXENGINX\Instrument\data\cycle_26_1

04/05/2026  19:35        22,353,920 ENGINX00374779.raw
               1 File(s)     22,353,920 bytes
               0 Dir(s)  6,511,410,020,352 bytes free

more < \\isis\inst$\NDXENGINX\Instrument\data\cycle_26_1\ENGINX00374779.raw:checksum
11188200fe269a312c43959361ab406c *ENGINX00374779.log
27e69f0786af731df812f6aa75fffcb3 *ENGINX00374779.n001
2fbf5c85a8778dbd479e9a507324f211 *ENGINX00374779.nxs
fc94181c28d8ca812e6982a58f51a1ec *ENGINX00374779.raw
875e53bfbfc8e76d4cd196ee3072b0bd *ENGINX00374779.s01
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_amplitude.txt
cb7175fc54d9d93fc1ec56afdf5c4bd0 *ENGINX00374779_C6_master_freq.txt
c3e4dbf1c60212fe28861fed361f3f67 *ENGINX00374779_C6_master_phase.txt
7fa12667088bb609165560b35bba0f2f *ENGINX00374779_C6_slave_freq.txt
0ab9f83b6e1e8063071500f1d3ddbd79 *ENGINX00374779_C6_slave_phase.txt
14aab4313c48ac51cc534fc34dcf71c9 *ENGINX00374779_C9_master_freq.txt
c1dbd0461bb22a154fb6fd964ee8b63d *ENGINX00374779_C9_master_phase.txt
85809e44e0fd798d96d348d8ebf6c253 *ENGINX00374779_C9_slave_freq.txt
8cd85cd45254de733c54ed86c0e250e9 *ENGINX00374779_C9_slave_phase.txt
c9c4c563c42e5b2086d5d9b3c55b76fa *ENGINX00374779_Chopper_mode.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_control_channel.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_control_channel_string.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_cycle_rbv.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_cycles.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_frequency.txt
909c82d1f9e7c10bc0046c33eed00d3c *ENGINX00374779_ICPalarm.txt
3f2760ed3b28dce0bbc056225e805964 *ENGINX00374779_ICPdebug.txt
8f6aa510eb964809a4b0ce04dcfae59e *ENGINX00374779_ICPevent.txt
85b0285d4a1e863a7d00033fbc7db215 *ENGINX00374779_ICPputlog.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_instron_logging.txt
82a64dacf2da5dc9c04ad90ccd5dcaeb *ENGINX00374779_jaws1_hgap.txt
ee0a1bf60a1455c0393a750a1d591a41 *ENGINX00374779_jaws1_vgap.txt
50e9f9b7383b6495243a9e2878dd51e2 *ENGINX00374779_jaws2_hgap.txt
59fc7e5aa7aa53c0a59efc2481184f01 *ENGINX00374779_jaws2_vgap.txt
18c236fa8e96e2b6343d5f3879215486 *ENGINX00374779_jaws3_hgap.txt
50add78d52d88729e900a5dcad8d7aef *ENGINX00374779_jaws3_vgap.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Max_output_1.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Max_output_2.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Max_output_3.txt
889863305a19e91d8646a36ebd3fd7a1 *ENGINX00374779_MethaneModTemp.txt
27312aa25cf462f4a9082fa97f589702 *ENGINX00374779_North_X.txt
532aeb150d963333592e5d5e66fff9ba *ENGINX00374779_North_X_hlim.txt
532aeb150d963333592e5d5e66fff9ba *ENGINX00374779_North_X_llim.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Output_1.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Output_2.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Output_3.txt
ece3792163af14497e5e23eae8b5c7ca *ENGINX00374779_Output_Rate_3.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_pos_ramp_wftyp.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_pos_step_time.txt
1ef09505c29399bd4e3ab1444ae6629e *ENGINX00374779_position.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_position_sp_rbv.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_Rampon_3.txt
5b4dd30ffb4e0a50cfc51125dea64f9d *ENGINX00374779_Ramprate_3.txt
e7334e923057d390b074f3d9e475c5c7 *ENGINX00374779_S_position.txt
f1e5ec2e01454bf02291b7f0adce6353 *ENGINX00374779_ShutterMode.txt
f48273934dd9106e2ed6ab77c1976378 *ENGINX00374779_South_X.txt
532aeb150d963333592e5d5e66fff9ba *ENGINX00374779_South_X_hlim.txt
532aeb150d963333592e5d5e66fff9ba *ENGINX00374779_South_X_llim.txt
1331731e9498cf55460c6528e23f93ff *ENGINX00374779_Status.txt
1ef09505c29399bd4e3ab1444ae6629e *ENGINX00374779_strain.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_strain_ramp_wftyp.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_strain_sp_rbv.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_strain_step_time.txt
1ef09505c29399bd4e3ab1444ae6629e *ENGINX00374779_stress.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_stress_ramp_wftyp.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_stress_sp_rbv.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_stress_step_time.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_stressrig_go.txt
0ed516a0155518b2f194c1fa078d7047 *ENGINX00374779_Temp_1.txt
5c1dbd612fc9e8be7c4723b48be504c6 *ENGINX00374779_Temp_2.txt
d0f929030b316698fcc471c284457191 *ENGINX00374779_Temp_3.txt
9724fb8b1dee0acff8f0ef4c3ddce126 *ENGINX00374779_W_position.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_wave_running.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_wave_start.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_wave_stop.txt
3f6f3fca03b002071e8741ac7eea6806 *ENGINX00374779_wave_type.txt
f4febb4e9a2f925b05058255d7edfcc9 *ENGINX00374779_X_position.txt
046d5d53d6d015fcd6abcafae294a9b2 *ENGINX00374779_Y_position.txt
ab4f1d9d7e494f61d10f445b2cc20e17 *ENGINX00374779_Z_position.txt
27e69f0786af731df812f6aa75fffcb3 *ENGINX00374779.n001
875e53bfbfc8e76d4cd196ee3072b0bd *ENGINX00374779.s01

[Tom-Willemsen]

Thanks @ChrisM-S - we do also have a more detailed issue for file archiving and checksumming strategy in the streaming system: ISISComputingGroup/DataStreaming#85 which you might be interested in.

We also have an issue to investigate how we want to do autosave/intermediate files (ISISComputingGroup/DataStreaming#96), which is of relevance to your comment about moving away from .n001 extensions for autosaves).

I've pushed an additional commit to add context about how these .raw files are used for checksumming.