Deep learning-based point-scanning super-resolution imaging

Linjing Fang; Fred Monroe; Sammy Weiser Novak; Lyndsey Kirk; Cara R. Schiavon; Seungyoon B. Yu; Tong Zhang; Melissa Wu; Kyle Kastner; Alaa Abdel Latif; Zijun Lin; Andrew Shaw; Yoshiyuki Kubota; John Mendenhall; Zhao Zhang; Gulcin Pekkurnaz; Kristen Harris; Jeremy Howard; Uri Manor

doi:10.1038/s41592-021-01080-z

Deep learning-based point-scanning super-resolution imaging

Linjing Fang
, Fred Monroe
, Sammy Weiser Novak
, Lyndsey Kirk
, Cara R. Schiavon
, Seungyoon B. Yu
, Tong Zhang
, Melissa Wu
, Kyle Kastner
, Alaa Abdel Latif
, Zijun Lin
, Andrew Shaw
, Yoshiyuki Kubota
, John Mendenhall
, Zhao Zhang
, Gulcin Pekkurnaz
, Kristen Harris
, Jeremy Howard
, Uri Manor

Research output: Contribution to journal › Article › peer-review

Abstract

Point-scanning imaging systems are among the most widely used tools for high-resolution cellular and tissue imaging, benefiting from arbitrarily defined pixel sizes. The resolution, speed, sample preservation and signal-to-noise ratio (SNR) of point-scanning systems are difficult to optimize simultaneously. We show these limitations can be mitigated via the use of deep learning-based supersampling of undersampled images acquired on a point-scanning system, which we term point-scanning super-resolution (PSSR) imaging. We designed a ‘crappifier’ that computationally degrades high SNR, high-pixel resolution ground truth images to simulate low SNR, low-resolution counterparts for training PSSR models that can restore real-world undersampled images. For high spatiotemporal resolution fluorescence time-lapse data, we developed a ‘multi-frame’ PSSR approach that uses information in adjacent frames to improve model predictions. PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed and sensitivity. All the training data, models and code for PSSR are publicly available at 3DEM.org.

Original language	American English
Pages (from-to)	406-416
Number of pages	11
Journal	Nature Methods
Volume	18
Issue number	4
DOIs	https://doi.org/10.1038/s41592-021-01080-z
State	Published - Apr 2021
Externally published	Yes

ASJC Scopus subject areas

Biotechnology
Biochemistry
Molecular Biology
Cell Biology

Access to Document

10.1038/s41592-021-01080-z

Cite this

Fang, L., Monroe, F., Novak, S. W., Kirk, L., Schiavon, C. R., Yu, S. B., Zhang, T., Wu, M., Kastner, K., Latif, A. A., Lin, Z., Shaw, A., Kubota, Y., Mendenhall, J., Zhang, Z., Pekkurnaz, G., Harris, K., Howard, J., & Manor, U. (2021). Deep learning-based point-scanning super-resolution imaging. Nature Methods, 18(4), 406-416. https://doi.org/10.1038/s41592-021-01080-z

@article{08abc98d27e943879c4fc416b67daf66,

title = "Deep learning-based point-scanning super-resolution imaging",

abstract = "Point-scanning imaging systems are among the most widely used tools for high-resolution cellular and tissue imaging, benefiting from arbitrarily defined pixel sizes. The resolution, speed, sample preservation and signal-to-noise ratio (SNR) of point-scanning systems are difficult to optimize simultaneously. We show these limitations can be mitigated via the use of deep learning-based supersampling of undersampled images acquired on a point-scanning system, which we term point-scanning super-resolution (PSSR) imaging. We designed a {\textquoteleft}crappifier{\textquoteright} that computationally degrades high SNR, high-pixel resolution ground truth images to simulate low SNR, low-resolution counterparts for training PSSR models that can restore real-world undersampled images. For high spatiotemporal resolution fluorescence time-lapse data, we developed a {\textquoteleft}multi-frame{\textquoteright} PSSR approach that uses information in adjacent frames to improve model predictions. PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed and sensitivity. All the training data, models and code for PSSR are publicly available at 3DEM.org.",

author = "Linjing Fang and Fred Monroe and Novak, \{Sammy Weiser\} and Lyndsey Kirk and Schiavon, \{Cara R.\} and Yu, \{Seungyoon B.\} and Tong Zhang and Melissa Wu and Kyle Kastner and Latif, \{Alaa Abdel\} and Zijun Lin and Andrew Shaw and Yoshiyuki Kubota and John Mendenhall and Zhao Zhang and Gulcin Pekkurnaz and Kristen Harris and Jeremy Howard and Uri Manor",

note = "Publisher Copyright: {\textcopyright} 2021, The Author(s), under exclusive licence to Springer Nature America, Inc.",

year = "2021",

month = apr,

doi = "10.1038/s41592-021-01080-z",

language = "American English",

volume = "18",

pages = "406--416",

journal = "Nature Methods",

issn = "1548-7091",

publisher = "Lippincott Williams and Wilkins Ltd.",

number = "4",

}

TY - JOUR

T1 - Deep learning-based point-scanning super-resolution imaging

AU - Fang, Linjing

AU - Monroe, Fred

AU - Novak, Sammy Weiser

AU - Kirk, Lyndsey

AU - Schiavon, Cara R.

AU - Yu, Seungyoon B.

AU - Zhang, Tong

AU - Wu, Melissa

AU - Kastner, Kyle

AU - Latif, Alaa Abdel

AU - Lin, Zijun

AU - Shaw, Andrew

AU - Kubota, Yoshiyuki

AU - Mendenhall, John

AU - Zhang, Zhao

AU - Pekkurnaz, Gulcin

AU - Harris, Kristen

AU - Howard, Jeremy

AU - Manor, Uri

PY - 2021/4

Y1 - 2021/4

N2 - Point-scanning imaging systems are among the most widely used tools for high-resolution cellular and tissue imaging, benefiting from arbitrarily defined pixel sizes. The resolution, speed, sample preservation and signal-to-noise ratio (SNR) of point-scanning systems are difficult to optimize simultaneously. We show these limitations can be mitigated via the use of deep learning-based supersampling of undersampled images acquired on a point-scanning system, which we term point-scanning super-resolution (PSSR) imaging. We designed a ‘crappifier’ that computationally degrades high SNR, high-pixel resolution ground truth images to simulate low SNR, low-resolution counterparts for training PSSR models that can restore real-world undersampled images. For high spatiotemporal resolution fluorescence time-lapse data, we developed a ‘multi-frame’ PSSR approach that uses information in adjacent frames to improve model predictions. PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed and sensitivity. All the training data, models and code for PSSR are publicly available at 3DEM.org.

AB - Point-scanning imaging systems are among the most widely used tools for high-resolution cellular and tissue imaging, benefiting from arbitrarily defined pixel sizes. The resolution, speed, sample preservation and signal-to-noise ratio (SNR) of point-scanning systems are difficult to optimize simultaneously. We show these limitations can be mitigated via the use of deep learning-based supersampling of undersampled images acquired on a point-scanning system, which we term point-scanning super-resolution (PSSR) imaging. We designed a ‘crappifier’ that computationally degrades high SNR, high-pixel resolution ground truth images to simulate low SNR, low-resolution counterparts for training PSSR models that can restore real-world undersampled images. For high spatiotemporal resolution fluorescence time-lapse data, we developed a ‘multi-frame’ PSSR approach that uses information in adjacent frames to improve model predictions. PSSR facilitates point-scanning image acquisition with otherwise unattainable resolution, speed and sensitivity. All the training data, models and code for PSSR are publicly available at 3DEM.org.

UR - https://www.scopus.com/pages/publications/85102253192

UR - https://www.scopus.com/pages/publications/85102253192#tab=citedBy

U2 - 10.1038/s41592-021-01080-z

DO - 10.1038/s41592-021-01080-z

M3 - Article

C2 - 33686300

SN - 1548-7091

VL - 18

SP - 406

EP - 416

JO - Nature Methods

JF - Nature Methods

IS - 4

ER -

Deep learning-based point-scanning super-resolution imaging

Abstract

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this