# Read and write files

!!! abstract “”

    How to use `open_()` for reading and writing files with automatic compression detection (gzip, bzip2, lzma, zip), `atomic_write` for safe file writes that clean up on failure, `iter_splitlines` and `iter_line_blocks` for streaming large files, and `is_url`/`open_url` for working with URLs.

## Writing a compressed file

`open_()` detects the compression format from the file suffix and handles it automatically. Writing a gzip-compressed text file is identical to writing a plain text file — just use a `.gz` suffix.

```python { linenums=“1” notest } from scinexus import open_

with open_(“data/sample.txt.gz”, “wt”) as f: f.write(“Hello, compressed world!”) f.write(“Line two of the file.”)

    <!-- [[[end]]] -->

    ## Reading a compressed file

    Reading works the same way — `open_()` detects the `.gz` suffix and decompresses transparently.

    <!-- [[[cog
    from cog_utils import exec_codeblock, DOCS_DIR

    src = \
    """
    from scinexus import open_

    with open_("data/sample.txt.gz") as f:
        print(f.read())
    """
    exec_codeblock(src=src)

    import pathlib
    pathlib.Path(DOCS_DIR / "data/sample.txt.gz").unlink(missing_ok=True)
      ]]] -->
    ```python { linenums="1" notest }
    from scinexus import open_

    with open_("data/sample.txt.gz") as f:
        print(f.read())

    # Hello, compressed world! Line two of the file.

Supported compression formats are gzip (`.gz`), bzip2 (`.bz2`), lzma (`.xz`, `.lzma`), and zip (`.zip`).

## Reading a URL

`open_()` also handles URLs. Use `is_url()` to check whether a path is a URL before opening it.

???- example “Checking and reading a URL”

    ```python linenums="1"
    from scinexus.io_util import is_url
    from scinexus import open_

    url = "https://github.com/user-attachments/files/26728407/raw.zip"
    print(is_url(url))  # (1)!

    with open_(url, "rb") as f:  # (2)!
        header = f.read(20)
    print(header)
    ```

    1. `is_url()` returns `True` for `http`, `https`, and `file` scheme URLs.
    2. `open_()` detects the URL and delegates to `open_url()`. Only read mode is supported for URLs.

## Efficiently reading large files

Reading an entire large file into memory or iterating line by line with Python’s built-in `readline()` can be inefficient. The built-in approach makes a system call for every line, which becomes a bottleneck for files with millions of lines. `scinexus` provides two functions that read data in large chunks and then split into lines, greatly reducing I/O overhead.

### `iter_splitlines`

`iter_splitlines(path, chunk_size=1_000_000)` reads a file in chunks (default 1 MB) and yields individual lines. It correctly handles lines that span chunk boundaries.

```python { notest } from scinexus.io_util import iter_splitlines

for line in iter_splitlines(“large_file.txt”): process(line)

    ### `iter_line_blocks`

    `iter_line_blocks(path, num_lines=1000, chunk_size=5_000_000)` builds on `iter_splitlines` — it accumulates lines into lists of `num_lines` and yields each list. This is useful when downstream processing works on batches (e.g. FASTA records where each record spans a fixed number of lines).

    ```python { notest }
    from scinexus.io_util import iter_line_blocks

    for block in iter_line_blocks("large_file.txt", num_lines=1000):
        process_batch(block)  # block is a list of up to 1000 strings

Use `iter_splitlines` when you need one line at a time. Use `iter_line_blocks` when your processing naturally operates on batches of lines.