Imager Platform Software

This repository contains the code for Imager v2 platform whose aim it is to enable massively parallel screening of certain properties of clusters of DNA, such as aptamer binding affinities.

Overview

The Imager v2 merges sequencing data from an Illumina Miseq sequencer and microscopy images captured on a modified Illumina GAII sequencer. The Illumina Miseq is run once to generate a flow cell. This flow cell is then taken and placed on the modified Illumina GAII for further experimentation. We can run the following types of operations on flow cells on the modified Illumina GAII:

• Flow different liquids through the flow cell (e.g., complementary strands, target molecules...).
• Control temperature (e.g., for annealing strands inside the flow cell).
• Take snapshot images of all tiles (we only image tiles 2101-2119) (green/red excitation, green/red emission).
• Take movies of one tile at a time.

To evaluate an Imager v2 experiment, we want the following information about every cluster on the flow cell:

• DNA sequence identity: What DNA sequence do the strands in a cluster of DNA have? This information comes from the .fastq files from the Illumina Miseq run.
• Intensity: How bright are the clusters in every image (either snapshots or every frame of a movie)

Steps of processing a run

To link sequence identity of a cluster with its intensity in an image, we need to merge the data coming off of the Illumina Miseq and the modified Illumina GAII. This process happens in multiple steps:

1. Sorting the records (one record contains information about a single cluster) in the .fastq files from the Illumina Miseq into sets. Usually, we want all the clusters in a set to light up together/in a similar way. For example, if we have a set of positive-control clusters that will always light up (we call these fiducial markers), we would want all of them to be in the same set. After this step we have sets of clusters that should behave similarly.
2. Aligning the cluster coordinates from the .fastq records with the coordinate system of the images coming from the modified Illumina GAII. After this step, for every image, we will have a ( non-linear) coordinate transformation that converts from the coordinate system used in the .fastq files to the coordinate system the image.
3. Calculating how bright every cluster reported in the .fastq records is in the images. This is done by looking at what coordinate a cluster land ons in the image after applying the coordinate transformation obtained in the previous step, and subsequently calculating the intensity of that location.

Installation

1. Install Python 3.
2. (Optional) Install git.
3. Download or clone this repository to your computer.
4. Create a virtual environment (for instructions using Conda, visit: https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html)
5. Activate the virtual environment in the terminal.
6. Validate that the virtual environment is activated: which python
7. Install the requirements: python -m pip install -r requirements.txt
8. Done!

Processing a Run

Before processing the data, we need to ensure that it is structured correctly. All files relating to the run should be in a folder whose path we will refer to as DATA_PATH from here on out. The folder structure inside of DATA_PATH should look like this:

• _metadata.csv: this file contains information about the Imager v2 run such as the names of the individual cycles and which folders they are in. Usually, this should be autogenerated by the instrument, but for some experiments, you will have to manually edit this file.
• fastq/: contains all the .fastq files generated by the Illumina Miseq.
  • FM_S1_L001_R1_001.fastq (Example file name)
  • Undetermined_S0_L001_R1_001.fastq (Example file name)
  • OTHER_FASTQ_FILES.fastq (Example file name)
• movies/: when the data to be analyzed is movie data.
  • movie_name_0/ (Example folder name)
    • 2101/ (images for movie of tile 2101)
      • GG_pTIRF_2101_x_-749004.4_y_-405624.1_t_1.tif (Example file name)
      • GG_pTIRF_2101_x_-749004.4_y_-405624.1_t_2.tif (Example file name)
      • ...
    • 2102/ (images for movie of tile 2102)
      • ...
    • ...
  • movie_name_1/ (Example folder name)
    • ...
  • ...
• ptirf/: when the data to be analyzed are snapshots.
  • Cycle_02_1/ (Example folder name)
    • GG_pTIRF_2101_x_-749373.5_y_-404215.2_t_1713455519.5827_z_-293041.7.tif (image of tile 2101)
    • GG_pTIRF_2102_x_-749346.2_y_-412507.2_t_1713455539.0806_z_-293045.2.tif (image of tile 2102)
    • ...
    • Cycle_03_1/ (Example folder name)
      • ...
    • Cycle_04_1/ (Example folder name)
      • ...
    • ...

Sorting records in the .fastq files into sets

1. Launch either GUI/GUI_main_img.py, or GUI/GUI_main_mov.py. This will open up a window that looks like this:
2. Hit the "Browse" button to select a folder. Important: You HAVE to use the button. If you just change the path in the text-field, the GUI does not take that into account.
3. Select an appropriate number of threads for you device. If your machine does not have a lot of RAM, it might be better to select a smaller number.
4. Hit the "Define FastQ Subsets" button. This may take a while to execute, but will eventually open a window looking something like this: Explanation of GUI:
   • At the top of the GUI, you have a list of all the FastQ files that were found in the /fastq folder in DATA_PATH/fastq. If this list is empty, it may be that you did not specify the right datapath in step 2, or you might not have decompressed the .fastq files (i.e., the file ending is still .fastq.gz). In this case, you have to decompress the .fastq files first, and restart the GUI.
   • The middle segment of the GUI in the gray box will contain the sets we are trying to define.
   • The bottom segment of the GUI with the buttons allows you to create a new set if you hit the '+' button, or it allows you to finalize your set definitions by hitting the 'Save and Close' button.
5. Hit the '+' button to create a new set. Explanation of the GUI:
   • The left-most '-' button allows you to delete a set.
   • The '↻' button allows you to re-calculate a set to see how many members it has.
   • The left-most text field contains the 'Name' of the set. Every set must have a distinct name.
   • The second text field contains the regular expression (=regex) that defines what a member sequence of this set should look like. If you are not familiar with regular expressions, there are many tutorials online (e.g., https://datasciencedojo.com/blog/regular-expression-101/).
   • The checkboxes allow you to select which of the .fastq files should be searched. The left-most checkbox selects the first .fastq file at the top of the GUI (In this example ' FM_S1_L001_R1_001'). The second checkbox selects the second .fastq file, and so on. If you know that your sequencing run had a lot of failed reads, it might be worth it to check the box for the undetermined clusters.
   • The numbers on the right correspond to the number of matched clusters per tile. The left-most number corresponds to tile 2101, the second number relates to 2102, etc. The last number on the far right is the sum of the preceding 19 columns and is equal to the total number of clusters in this set.
6. Define the set. In this example, we are defining the fiducial marker set, which contains all sequences that begin with 'AACCAGG'. The corresponding regex is ^AACCAGG[ATCGN]*$. After hitting the '↻' button, the GUI looks like this: We see that we have around 700-800 clusters per tile, and that the total number of clusters (last column) is around 14,000.
7. Add further sets as desired. Usually, you will want one set for fiducial markers, one for undetermined clusters, and one for every cluster family you loaded on the flow cell:
8. Hit the "Save and Close" button. This will take a minute to complete.

Generate coordinate transformations from .fastq coordinates to image coordinates

Having sorted our clusters with similar sequences in the .fastq into sets in the previous step, we can now use this information to align the images with every image we took.

1. Launch either GUI/GUI_main_img.py, or GUI/GUI_main_mov.py. This will open up a window that looks like this:
2. Hit the "Browse" button to select a folder. Important: You HAVE to use the button. If you just change the path in the text-field, the GUI does not take that into account.
3. Select an appropriate number of threads for you device. If your machine does not have a lot of RAM, it might be better to select a smaller number.
4. Hit the "Peak Detection Alignment" button. This will launch the following GUI: Explanation of GUI:
   • Top of GUI: Configuration options
     • Number of threads: how many threads will be launched to process the data.
     • Alignment cutoff: how good the alignment needs to be acceptable. The unit of alignment is pixels. It is measured as the average of the cluster alignment distances that are left after accounting for random/spurious alignments.
     • GG/RR/GR default: default peak detection parameters for each color channel.
       • Sigma Low Pass [px]: Standard deviation of the low-pass filter that is applied to the images before detecting peaks. Higher number means more information is removed because we blur more.
       • Sigma High Pass [px]: Standard deviation of the high-pass filter that is applied to the images before detecting peaks. Lower number means more information is removed because we only keep very sharp peaks.
       • Cutoff Threshold [RFU]: Cutoff intensity after low/high-pass filtering below which a peak will not be counted as a peak. Higher number means we are more stringent about how bright a peak has to be.
   • Middle of GUI: Each cycle gets its own row. From left to right, each row contains the following information about the cycle.
     • Folder name: in this case "Cycle_02_1". The folder in which the images are found
     • Cycle name: in this case "img_0". The colloquial name of what experimental condition this cycle represents.
     • Custom Cutoff Threshold [RFU]: In case you want to define a different cutoff threshold for peak detection for just one cycle, you can do so here.
     • The two '↻' buttons.
       • Top '↻' button: runs peak detection for just this cycle.
       • Bottom '↻' button: runs coordinate alignment for just this cycle.
     • Edit FastQ Sets: Allows you to define which sets to use to attempt alignment. It is a good idea to only use those sets that you expect to be bright in this experimental condition. For example, if you only expect the fiducial marker clusters to be bright in a given experimental condition, you should only select the FM set.
     • Cycle table:
       • Top row: tile numbers
       • Middle row: number of clusters detected in every tile's image.
       • Bottom row: Alignment scores for every tile.
     • "Generate Coordinates File" Button: If the alignments all look good, you can press this button to generate a coordinates.csv file that can be used by the modified Illumina GAII to take images at the correct location.
   • Bottom of the GUI:
     • "Run Peak Extraction for All": runs peak extraction for all cycles.
     • "Run Alignment for All": runs alignment for all cycles.
     • "Run Both for All": runs both peak extraction and alignment for all cycles.
     • "Save and Close": Saves the settings you selected and closes the GUI safely.
5. Hit the top '↻' button of a row to run peak detection. This will populate the middle row of the table with numbers indicating how many clusters were detected in each image:
6. Click on one of the newly populated numbers to open up a window that lets you visually inspect the image and the detected peaks: In this case, it looks like the brightest peaks were detected well (as marked by red crosses). If too many/too few peaks were detected, consider changing the "Cutoff Threshold [RFU]" value and starting again from step 5.
7. Once you are happy with the detected peaks, hit the "Edit FastQ Sets" button. This will open up a GUI that will allow you to define which FastQ Set should be aligned with the peaks we just detected. In this case, we only want to align with the fiducial markers because they are the only clusters we expect to be bright. Hit save and close once you are happy with your selection.
8. Hit the bottom '↻' button of a row to run coordinate alignment. This will populate the lowest row in the table of the row with numbers representing how good the alignment was. If alignment score is worse that the "Alignment Cutoff", the number will be displayed in red.
9. Click on the newly populated numbers to open up a window that lets you visually inspect the image and the aligned coordinates. In the zoomed out picture, we can see the peaks that were detected in the image in red, and the aligned coordinates from the .fastq files in blue. In this example, we can see that the blue aligned coordiantes are not well-centered with the image. Once we zoom in, we can verify if the coordinate transformation looks good: In this case, the coordinate alignment looks good, because all the blue aligned coordinates line up with cluster in the image.
10. Once we are happy with this cycle, we can hit "Run Both for All" to run peak detection and alignment for all cycles. Upon successful completion, we can hit "Save and Close".

Intensity Extraction

Once we have aligned the .fastq coordinates with all the images, we can extract intensities.

1. Launch either GUI/GUI_main_img.py, or GUI/GUI_main_mov.py. This will open up a window that looks like this:
2. Hit the "Browse" button to select a folder. Important: You HAVE to use the button. If you just change the path in the text-field, the GUI does not take that into account.
3. Select an appropriate number of threads for you device. If your machine does not have a lot of RAM, it might be better to select a smaller number.
4. Hit the "Intensity Extraction" button.

Structure of _metadata.csv file

See /templates/_metadata.csv

Key,Value,Note
EXP_DATE,2024-06-12,Use ISO format YYYY-MM-DD
EXP_NAME,NAME_OF_YOUR_EXPERIMENT,name of experiment
EXP_PERSON,YOUR_NAME,person who performed the experiment
SEQUENCER,miseq,options:miseq
SEQUENCER_FLOWCELL,miseqv3,options:miseqv1/miseqv2/miseqv3
IMAGER,ptirf2,options:ga2/ti2/miseq
IMAGER_IMAGE_TYPE,,only for miseq-miseq; which of the color channels to use (a;t;c;g)
IMAGER_CHANNELS,GG,for miseq-ga2; which color channels to process (GG;GR;RR)
IMAGER_FOLDER_NAMES_MOV,MOVIE;FOLDER;NAMES;SEPARATED;BY;SEMICOLONS,names of the folders in the CONFIG.DIR_IMG folder (in same order as IMAGER_CYCLE_ORDER)
IMAGER_CYCLE_ORDER_MOV,TYPE;OF;EACH;MOVIE;FOLDER;SEPARATED;BY;SEMICOLONS,order of imager cycles (give the type for each cycle. The program will add the indices)
IMAGER_TILT,,Usually blank but can be useful when you had a really badly aligned flow cell clamp
IMAGER_FOLDER_NAMES,IMAGE;FOLDER;NAMES;SEPARATED;BY;SEMICOLONS,names of the folders in the CONFIG.DIR_IMG folder (in same order as IMAGER_CYCLE_ORDER)
IMAGER_CYCLE_ORDER,TYPE;OF;EACH;IMAGE;FOLDER;SEPARATED;BY;SEMICOLONS,order of imager cycles (give the type for each cycle. The program will add the indices)