Skip to main content
`Ctrl`+`K`

# Obtaining subsets from cloud-hosted FITS files

Astropy offers support for extracting data from FITS files stored in the cloud. Specifically, the `astropy.io.fits.open` function accepts the `use_fsspec` and `fsspec_kwargs` parameters, which allow remote files to be accessed in an efficient way using the |fsspec| package.

`fsspec` is an optional dependency of Astropy which supports reading files from a range of remote and distributed storage backends, such as Amazon and Google Cloud Storage. This chapter explains its use.

Note

The examples in this chapter require `fsspec` which is an optional dependency of Astropy. See Installation for details on installing optional dependencies.

## Subsetting FITS files hosted on an HTTP web server

A common use case for `fsspec` is to read subsets of FITS data from a web server which supports serving partial files via the Range Requests feature of the HTTP protocol. Most web servers support serving portions of files in this way.

For example, let’s assume you want to retrieve data from a large image obtained by the Hubble Space Telescope available at the following url:

    >>> # Download link for a large Hubble archive image (213 MB)
    >>> url = "https://mast.stsci.edu/api/v0.1/Download/file/?uri=mast:HST/product/j8pu0y010_drc.fits"

This file can be opened by passing the url to `astropy.io.fits.open`. By default, Astropy will download the entire file to local disc before opening it. This works fine for small files but tends to require a lot of time and memory for large files.

You can improve the performance for large files by passing the parameter `use_fsspec=True` to `open`. This will make Astropy use `fsspec` to download only the necessary parts of the FITS file. For example:

    >>> from astropy.io import fits
    ...
    >>> # `fits.open` will download the primary header
    >>> with fits.open(url, use_fsspec=True) as hdul:
    ...
    ...     # Download a single header
    ...     header = hdul[1].header
    ...
    ...     # Download a single data array
    ...     image = hdul[1].data
    ...
    ...     # Download a 10-by-20 pixel cutout by using .section
    ...     cutout = hdul[2].section[10:20, 30:50]

The example above requires less time and memory than would be required to download the entire file. This is because `fsspec` is able to leverage two lazy data loading features available in Astropy:

1. The `lazy_load_hdus` parameter offered by `open` takes care of loading HDU header and data attributes on demand rather than reading all HDUs at once. This parameter is set to `True` by default. You do not need to pass it explicitly, unless you changed its default value in the Configuration System (astropy.config).

2. The `ImageHDU.section` and `CompImageHDU.section` properties enables a subset of a data array to be read into memory without downloading the entire image or cube. See the Data Sections part of the documentation for more details.

Additional tips for achieving good performance when working with remote files are provided in the Performance improvement tips for subsetting remote FITS files section further down this page.

Note

The `ImageHDU.section` and `CompImageHDU.section` feature is only efficient for files that are not externally compressed (such as `.fits.gz` files). Files that are compressed using internal tile compression should work properly. Use `.section` on an externally compressed image will cause the whole FITS file to be downloaded.

## Subsetting FITS files hosted in Amazon S3 cloud storage

The FITS file used in the example above also happens to be available via Amazon cloud storage, where it is stored in a public S3 bucket at the following location:

    >>> s3_uri = "s3://stpubdata/hst/public/j8pu/j8pu0y010/j8pu0y010_drc.fits"

With `use_fsspec` enabled, you can obtain a small cutout from a file stored in Amazon S3 cloud storage in the same way as above. When opening paths with prefix `s3://` (Amazon S3 Storage) or `gs://` (Google Cloud Storage), `open` will automatically default to `use_fsspec=True` for convenience. For example:

    >>> # Download a small 10-by-20 pixel cutout from a FITS file stored in Amazon S3
    >>> with fits.open(s3_uri, fsspec_kwargs={"anon": True}) as hdul:
    ...     cutout = hdul[1].section[10:20, 30:50]

Obtaining cutouts from Amazon S3 in this way may be particularly performant if your code is running on a server in the same Amazon cloud region as the data.

Note

To open paths with prefix `s3://`, fsspec requires an optional dependency called |s3fs|. A `ModuleNotFoundError` will be raised if this dependency is missing. See Installation for details on installing optional dependencies.

### Working with Amazon S3 access credentials

In the example above, we passed `fsspec_kwargs={"anon": True}` to enable the data to be retrieved in an anonymous way without providing Amazon cloud access credentials. This is possible because the data is located in a public S3 bucket which has been configured to allow anonymous access.

In some cases you may want to access data stored in an Amazon S3 data bucket that is private or uses the “Requester Pays” feature. You will have to provide a secret access key in this case to avoid encountering a `NoCredentialsError`. You can use the `fsspec_kwargs` parameter to pass extra arguments, such as access keys, to the `fsspec.open` function as follows:

    >>> fsspec_kwargs = {"key": "YOUR-SECRET-KEY-ID",
    ...                  "secret": "YOUR-SECRET-KEY"}
    >>> with fits.open(s3_uri, fsspec_kwargs=fsspec_kwargs) as hdul:
    ...     cutout = hdul[2].section[10:20, 30:50]

Warning

Including secret access keys inside Python code is dangerous because you may accidentally end up revealing your keys when you share your code with others. A better practice is to store your access keys via a configuration file or environment variables. See the |s3fs| documentation for guidance.

## Using `Cutout2D` with cloud-hosted FITS files

The examples above used the `ImageHDU.section` feature to download small cutouts given a set of pixel coordinates. For astronomical images it is often more convenient to obtain cutouts based on a sky position and angular size rather than array coordinates. For this reason, Astropy provides the `astropy.nddata.Cutout2D` tool which makes it easy to obtain cutouts informed by an image’s World Coordinate System (`WCS`).

This cutout tool can be used in combination with `fsspec` and `.section`. For example, assume you happen to know that the image we opened above contains a nice edge-on galaxy at the following position:

    >>> # Approximate location of an edge-on galaxy
    >>> from astropy.coordinates import SkyCoord
    >>> position = SkyCoord('10h01m41.13s 02d25m20.58s')

We also know that the radius of the galaxy is approximately 5 arcseconds:

    >>> # Approximate size of the galaxy
    >>> from astropy import units as u
    >>> size = 5*u.arcsec

Given this sky position and radius, we can use `Cutout2D` in combination with `use_fsspec=True` and `.section` as follows:

    >>> from astropy.nddata import Cutout2D
    >>> from astropy.wcs import WCS
    ...
    >>> with fits.open(s3_uri, use_fsspec=True, fsspec_kwargs={"anon": True}) as hdul:
    ...     wcs = WCS(hdul[1].header)
    ...     cutout = Cutout2D(hdul[1].section,  # use `.section` rather than `.data`!
    ...                       position=position,
    ...                       size=size,
    ...                       wcs=wcs)

See 2D Cutout Images for more details on this feature.

## Performance improvement tips for subsetting remote FITS files

In the examples above we explained that it is important to use the `use_fsspec=True` feature in combination with the `lazy_load_hdus=True` parameter and the `ImageHDU.section` feature to obtain good performance.

There are two additional factors which significantly impact the performance you will encounter, namely: (i) the structure of the FITS file, and (ii) the caching and block size configuration of `fsspec`. The remainder of this section briefly explains these two factors.

### Matching the FITS file structure to the data slicing patterns

The order in which multi-dimensional data is organized inside FITS files plays a major role in the subsetting performance.

Astropy uses the row-major order for indexing FITS data. This means that the right-most axis is the one that varies the fastest inside the file. Put differently, the data for the right-most dimension tends to be located in contiguous regions of the file and is therefore the easiest to extract.

For example, in the case of a 2D image, the slice `.section[0, :]` can be obtained by downloading one contiguous region of bytes from the file. In contrast, the slice `.section[:, 0]` requires accessing bytes spread across the entire image array. The same is true for higher dimensions, for example, obtaining the slice `.section[0, :, :]` from a 3D cube will tend to be much faster than requesting `.section[:, :, 0]`.

Obtaining slices of data that are well matched to the internal layout of the FITS file generally yields the best performance. If subsetting performance is important to you, you may have to consider modifying your FITS files to ensure that the ordering of the dimensions is well-matched to your data slicing patterns.

### Configuring the `fsspec` block size and download strategy

The `fsspec` package supports different data reading and caching strategies which aim to find a balance between the number of network requests on one hand and the total amount of data transferred on the other hand. By default, `fsspec` will attempt to download data in large contiguous blocks using a buffered read ahead strategy, similar to the strategy that is employed when operating systems load local files into memory.

You can tune the performance of `fsspec`’s buffering strategy by passing custom `block_size` and `cache_type` parameters to `fsspec.filesystem`, and passing the filesystem into the `fsspec_filesystem` keyword argument in `astropy.io.fits.open`. For example, we can configure fsspec to make buffered reads with a minimum `block_size` of 1 MB as follows:

    >>> import fsspec
    >>> fsspec_fs = fsspec.filesystem(
    ...     protocol="s3",
    ...     anon=True,
    ...     block_size=1_000_000,
    ...     cache_type="bytes"
    ... )
    >>> with fits.open(
    ...     s3_uri, fsspec_filesystem=fsspec_fs, fsspec_kwargs={"anon": True}
    ... ) as hdul:
    ...     cutout = hdul[1].section[10:20, 30:50]

The ideal configuration will depend on the latency and throughput of the network, as well as the exact shape and volume of the data you seek to obtain.

See the |fsspec| documentation for more information on its options.