# Changelog

## Changes in v0.8.0

This version further improves testing, improves the way C++ files are included in the package, and fixes many long-standing bugs.

- Using pytest framework rather than nose for testing
- Updated `setup.py` to be more robust and to more clearly separate “cythonization” into .cpp files
- Updated test harness for testing in independent conda environments
- All issue tests go in their own test module
- Included Python 3.7 tests (note that at the time of this writing, pysam is not yet available on bioconda so that dependency is pip-installed in the test) (#254)
- Updated tests to reflect BEDTool 2.27.1 output (#260 <https://github.com/daler/pybedtools/issues/260>`_`#261)
- Removed the `contrib.classifier` module, which has been unsupported for a while.
- More informative error messages for UCSC tools if they’re missing (#227)
- BedTool objects that are the result of operations that create files that are not BED/GTF/GFF/BAM can be more easily converted to pandas.DataFrame with `disable_auto_names=True` arg to `BedTool.to_dataframe()` (#258)
- Added aliases to existing methods to match current BEDTools commands, e.g. the `BedTool.nucleotide_content` method can now also be called using `BedTool.nuc` which is consistent with the `bedtools nuc` command line name.
- New wrapper for `bedtools split`. The wrapper method is called `splitbed` to maintain backwards compatibility because `pybedtools.BedTool` objects have long had a `split` method that splits intervals based on a custom function.
- New wrapper for `bedtools spacing`.
- `BedTool.from_dataframe` handles NaN in dataframes by replacing with `"."`, and is more explicit about kwargs that are passed to `pandas.DataFrame` (#257)
- Raise FileNotFoundError when on Python 3 (thanks Gosuke Shibahara, (#255)
- Relocated BEDTools header and .cpp files to the `pybedtools/include` directory, so they can more easily be linked to from external packages (#253)
- Add test for (#118)
- `BedTool.tabix_contigs` will list the sequence names indexed by tabix (#180)
- `BedTool.tabix_intervals` will return an empty generator if the coordinates provided are not indexed, unless `check_coordinates=True` in which case the previous behavior of raising a ValueError is triggered (#181)
- Bugfix: Avoid “ResourceWarning: unclosed file” in `helpers.isBGZIP` (thanks Stephen Bush)
- Bugfix: Interval objects created directly no longer have their filetype set to None (#217)
- Bugfix: Fixed the ability to set paths and reload module afterwards (#218, #220, #222)
- Bugfix: `BedTool.head()` no longer uses an IntervalIterator (which would check to make sure lines are valid BED/GTF/GFF/BAM/SAM). Instead, it simply prints the first lines of the underlying file.
- Bugfix: functions passed to `BedTool.filter` and `BedTool.each` no longer silently pass ValueErrors (#231)
- Bugfix: Fixed IndexError in IntervalIterator if there was an empty line (#233)
- Bugfix: Add additional constraint to SAM file detection to avoid incorrectly detecting a BED file as SAM (#246)
- Bugfix: accessing Interval.fields after accessing Interval.attrs no longer raises ValueError (#246)

## Changes in v0.7.10

Various bug fixes and some minor feature additions:

- Support for comma-separated lists for `mapBed` (thanks Chuan-Sheng Foo)
- Support many calls to `tabix_intervals` without hitting a “Too many open files” error (#190)
- Clarify documentation for `genome_coverage` when used with default parameters (#113)
- Ignore stderr from samtools on older zlib versions (#209, thanks Gert Hulselmans)
- Support fetching all regions from a chromosome (#201, thanks Matt Stone)
- Add wrapper for `shiftBed` (#200, thanks Saket Choudhary)
- Fix `truncate_to_chrom` in Python 3 (#203, thanks Saulius Lukauskas)
- When making bigWigs, use `bedSort` to ensure the sort order matches that expected by UCSC tools (#178)
- Fix newline handling of `pysam.ctabix.TabixIterator` output (#196)

## Changes in v0.7.9

Minor bugfix release:

- add `contrib.bigwig.bigwigtobedgraph` (thanks Justin Fear)
- fix `BedTool.seq()` in Python 3
- fix intron creation (#182, thanks @mmendez12)
- add `six` as an explicit requirement (#184, thanks @jrdemasi)
- improvements to setup (https://github.com/daler/pybedtools/issues/185)
- make pandas fully optional

## Changes in v0.7.8

- Be more careful about BAM vs bgzipped files (#168)
- `BedTool.bgzip` now preserves the header when sorting
- In Python 3, parsed BEDTools help string is decoded properly
- Ensure integer number of processes in Python 3 (thanks Illa Shamovsky)
- Add details on IOError messages for broken pipe error
- Make converting to pandas.DataFrames easier with non-standard BED files (thanks Panos Firmpas)

## Changes in v0.7.7

- Chromsizes for dm6 and mm10 assemblies added to `genome_registry`
- Better Python 3 compatibility in the `long_range_interaction` module
- New `featurefuncs.UniqueID` class, useful for ensuring all features in a file have a unique ID in their name field.
- Fix error message when a specified genome file doesn’t exist (thanks Saket Choudhary)

## Changes in v0.7.6

- New module `pybedtools.contrib.long_range_interaction` for working with HiC-like data.

## Changes in v0.7.5

- When using tabix-indexed files, `tabix` and `bgzip` are no longer required to be installed separately. Only `pysam` is needed.
- Recent BEDTools releases support multiple files for the `-b` argument of `bedtools intersect`. This version of `pybedtools` now supports multiple files as well. Note that it is still possible to provide a list of strings representing intervals as the `b` argument to `BedTool.intersect`. To differentiate between a list of intervals and a list of filenames, the first item converted into an `Interval` object; if it fails then consider the items to be filenames; otherwise assume strings representing intervals. This check only occurs if the `b` argument is a list or tuple; other iterable types are always assumed to be intervals.

## Changes in v0.7.4

Bugfix release.

- fix #147 so that warnings are simply passed to the user without raising exceptions
- in setup.py, allow depedencies to have “loose” versions with suffixes like “rc1”
- fix in `BedTool.cat()` on empty files (thanks Brad Chapman (PR #149)

## Changes in v0.7.1

This is largely a bugfix release with the following changes:

- fix for some BAM headers (thanks Gabriel Platt)
- unified IntervalIterator to address some streaming issues (fixes #143)
- fix bug where `__add__` was not re-raising exceptions (thanks Brad Chapman and Dan Halligan)

## Changes in v0.7.0

This release reflects a major upgrade in the underlying code in order to support both Python 2 and Python 3 using the same code. Aside from trivial things like converting print statements to functions and using `next()` instead of `.next()`, this required a substantial rewrite to support the way strings are handled in Python 3 (in Cython and wrapped C++) and how relative modules work.

Importantly, after converting them to Python 2- and 3-compatible syntax all previous tests pass so to the end user should not notice any differences except those noted below.

### Strings from Interval fields are unicode

For consistency between Python 2 and 3, all strings from Interval objects are now unicode. That is, in Python 2, previously we would get this:

    >>> a = pybedtools.example_bedtool('a.bed')
    >>> a[0].name
    'feature1'

Now, we get this:

    >>> a = pybedtools.example_bedtool('a.bed')
    >>> a[0].name
    u'feature1'

### samtools no longer a dependency

The dependency for samtools has been removed, which simplifies the installation process. Instead, `pysam` is used for handling BAM files.

In order for existing tests to pass, `pysam.AlignedSegment` objects are currently converted to `pybedtools.Interval` objects when iterating over a BAM file. This will come at a performance cost if you are iterating over all reads in a BAM file using the `pybedtools.BAM` object.

In the future, iterating over a BAM file will yield `pysam.AlignedSegment` objects directly, but for now you can use the `pybedtools.BAM.pysam_bamfile` attribute to access the underlying `pysam.AlignmentFile`

### Cython no longer a dependency

The Cythonized `.cxx` files are now shipped with the `pybedtools` distribution, so Cython is no longer a requirement for installation.

You will however need to have Cython installed if you’re developing pybedtools.

### Remote BAM support clarification

Previously, `pybedtools` was able to support remote BAMs by loosely wrapping samtools, but BAM files still needed to be fully downloaded to disk before using with BEDTools. This was done automatically, but through an inefficient mechanism.

Pysam does support remote BAMs, and as before, a BAM file needs to be created on disk for use with BEDTools. But now this needs to be explicitly done by the user, which should result in better performance.

### Iterating over intervals

Previously, when iterating over a `BedTool` object, different machinery would be invoked depending on whether the BedTool was pointing to a file (a cbedtools.IntervalFile would be invoked), to another iterator of Interval objects, or to a stream like from the stdout of a BEDTools call (cbedtools.IntervalIterator in both cases).

Everything is now an IntervalIterator, simplifying the path towards performance optimization.

### gzip support

Thanks to Saulius Lukauskas, gzip handling is now improved, and calling `BedTool.saveas()` with a `.gz` extension on the filename will automatically compress the output.

### Docker

In the github repo there is a `docker` directory containing Dockerfiles to set up isolated testing environments. These Dockerfiles also demonstrate how to set up a complete environment starting from a base Ubuntu install.

### Tests

All tests from v0.6.9 (which was Python 2 only) have been made Python 2/3 compatible and all previous tests pass.

If you have docker installed, from the top level directory, you can run the full tests like this:

    cd docker
    ./full-tests.sh

This will build docker containers for Python 2 and Python 3 with all depedencies, export the parent directory to the container, and run the test suite.

### Conda packages

You can now install the latest versions of tabix, bedtools, pysam, and pybedtools from conda, dramatically speeding up installation time. These mechanisms are used for automated testing as well (see the `condatest.sh` script in the github repo).

To use these packages in your own environment(s), specify the `daler` conda channel like this:

    conda install -c daler pybedtools

Note that this will not install BEDTools or tabix unless you explicitly say so:

    conda install -c daler pybedtools bedtools tabix

Note

This currently only works on Linux; contributions to Mac conda recipes (see the `conda` dir in the github repo) would be welcomed.

## Changes in v0.6.9

Minor bug fix release.

- improved the automatic field name handling when converting an interval file to a `pandas DataFrame`.
- fixed a bug in `IntervalFile` methods `all_hits`, `any_hits` and `count_hits` where zero-length features were being counted multiple times (thanks Brent Pedersen and Kyle Smith)
- bgzip and tabix paths can now be configured separately (thanks Rob Beagrie)
- fixed a bug where streaming BAM files were read fully into memory (thanks Alexey Sergushichev)

## Changes in v0.6.8

Bugfix: Thanks to Gabriel Pratt, `pybedtools` is no longer plagued by open filehandles in the C code causing the notorious “Too many files open” error.

## Changes in v0.6.7

Now compatible with BEDTools v2.21.0.

The one exception is that the new `bedtools intersect` functionality that allows multiple `-b` files is not yet implemented in `pybedtools`.

New features:

- `BedTool.fisher()` wraps the new BEDTools `fisher` tool. The result is an object containing parsed results.
- `BedTool.colormap_normalize()` accepts a `percentile` argument, useful when applying colormaps to data with a handful of extreme outliers
- `BedTool.to_datafame()` converts a `BedTool` object into a `pandas.DataFrame` with columns named after the appropriate fields for the filetype (thanks Radhouane Aniba for the suggestion)
- `BedTool.tail()` to complement `BedTool.head()` (thanks Radhouane Aniba for the suggestion)
- Add hg38 and hg38.default chromsizes

Minor bug fixes:

- Ensure tuple-like args to `parallel_apply` (fixes #109)
- Temp fix for BEDTools v2.20.0 which required the `-w` arg to come before the `-s` arg in `bedtools makewindows` (#81)
- Better (i.e., UCSC Genome Browser-compliant) defaults for `featurefuncs.expand_fields`.
- Fix for BedTool.all_hits() and any_hits() which will now show hits for zero-length features intersecting with other zero-length features with the same coordinates.

## Changes in v0.6.6

This is a compatibility release, updated for BEDTools v2.20.0.

There is one API change that affects the behavior of overloaded operators (that is, using `+` and `-` with BedTool objects) when one of the BedTool objects represents an empty file.

Assume `a` is a BedTool object representing a regular BED file but `b` is empty. Previously:

> - a + b = a
> - b + a = b
> - a - b = a
> - b - a = a
> - b - b = b
> - a + a = a

The following changes have been made (indicated in bold), which hopefully make more logical sense:

> - a + b = b
> - b + a = b
> - a - b = a
> - b - a = b
> - b - b = b
> - a + a = a

## Changes in v0.6.5

This is a minor bug-fix release:

- Fix for BedTool.all_hits() and any_hits() which will now show hits for zero-length features with the same coordinates, like the rest of BEDTools.
- Improved error-handling to avoid Python interpreter crashing in cases when a BED file on the filesystem becomes unavailable after a BedTool object has been created for it.

## Changes in v0.6.4

- Full integration with BEDTools v2.18. This includes some compatibility fixes for the new buffered output capabilities of BEDTool `intersect` and wrapping the new `bedtools sample` tool.
- Overloaded operators (`+` and `-`) allow empty files as input, even using BEDTools v2.18+.
- Travis-CI builds now use BEDTools v2.18+ for tests.
- Fix for `pybedtools.featurefuncs.midpoint()` (thanks ny-shao)
- Fix to `BedTool.randomstats()` (thanks Michael Reschen)

## Changes in v0.6.3

- New `pybedtools.parallel` module for working with many operations in parallel. See the docs for `pybedtools.parallel.parallel_apply()` for details.
- `pybedtools.contrib.bigbed.bigbed()` for converting to bigBed format, along with auto-SQL creation as needed.
- New function `pybedtools.contrib.bigbed.bigbed_to_bed()`, so now bigBed -> BED and BED -> bigBed interconversions are trivial.
- Support for remote BAMs by passing `remote=True` when creating a `BedTool` object
- New method `BedTool.at()` for subsetting a BedTool by a set of (sorted) indexes.
- New functions `featurefuncs.gff2bed()` and `featurefuncs.bed2gff()` for use with `BedTool.each()`, for easy converting GFF/GTF to BED
- New function `add_color()` for applying matplotlib colormaps to BED files; see also new method `pybedtools.BedTool.colormap_normalize()`.
- `pybedtools.plotting.BinaryHeatmap` class for working with results from `BedTool.multi_intersect()`.
- `BedTool.each()` now also has some filter capabilities (if provided function’s return value evaluates to False, feature will be skipped)
- Better detection for samtools (thanks Luca Beltrame)
- Expand BEDToolsError (thanks Ryan Layer)
- Creating a BedTool from a list of intervals now saves to temp file instead of treating like a consume-once iterator (#73)
- Various fixes to keyword arg handling to match semantics of BEDTools.
- Command line help and improved docs for the `peak_pie.py` script.
- Fix to GFF attributes (thanks Libor Mořkovský)
- Fix to labels in `pybedtools.contrib.venn_maker.py` (thanks Luca Pinello)
- Make the naive scaling (to million mapped reads) in `pybedtools.contrib.bigwig.bam_to_bigwiq()` optional.
- Fix for `BedTool.cat()` to handle cases where at least one input is an empty file
- Removed SciPy dependency
- Every commit is built with Travis-CI for continuous integration testing of changes to source code.

## Changes in v0.6.2

- Wrapped new tools available in BEDTools 2.17: `BedTool.jaccard()` and `BedTool.reldist()` wrap the new `bedtools jaccard` and `bedtools reldist` respectively.
- Initial implementations of building blocks for computing statistics, `BedTool.absolute_distance()` and `BedTool.relative_distance()`
- `pybedtools.featurefuncs.three_prime()`, `pybedtools.featurefuncs.five_prime()`, and `pybedtools.featurefuncs.TSS()` modifier functions that can be passed to `BedTool.each()`
- `pybedtools.contrib.plotting.binary_heatmap()` for visualizing results from `BedTool.multi_intersect()`
- Fixed a long-standing issue where streaming `BedTool` objects did not close their open file handles (stdout). When working with many (i.e. tens of thousands) files, this caused the operating system to hit its open file limit. This is now fixed.
- `BedTool.random_op()`, a new mechanism for implementing operations that you would like to apply over tens of thousands of shuffled interval files. This makes it easy to extend the existing `pybedtools` multiprocessing functionality.
- `pybedtools.contrib.bigwig.bam_to_bigwig()`, a helper function to create a libary-size-scaled bigWig file from an input BAM file.
- `pybedtools.contrib.plotting.TrackCollection` class, which handles plotting multiple files at once, using a provided “stylesheet” configuration to tweak colors etc.
- `pybedtools.contrib.plotting.BedToolsDemo` and `pybedtools.contrib.plotting.ConfiguredBedToolsDemo`, useful for running many graphical demos of BEDTools operations using the same “stylesheet” configuration. Run `pybedtools/contrib/plotting.py` for a demo.
- chromsizes dictionaries for common assemblies now have a `default` attribute, which is an OrderedDict of a default set of chromosome. For example, `pybedtools.chromsizes('hg19').default` contains only the entries for the autosomes and X and Y.
- `BedTool.cat()` now works better with multiprocessing
- added `include_distribution` kwarg to `BedTool.randomstats()`, which will attach the full distribution of all the randomized files to the results dictionary.
- New method implementing Jaccard statistic (with pvalue using randomizations): `BedTool.random_jaccard()`
- `featurefuncs.extend_fields()` helper function to pad fields with `'.'`, useful for manipulating features with the `BedTool.each()` method
- Fixed a bug where BAM files, when written to disk via `BedTool.saveas()`, were saved as SAM files.
- Better GTF/GFF detection, and if the input had quoted attribute values, then the output will, too
- various minor bug fixes and improvments as documented in the github commit logs….

## Changes in v0.6.1

- New `pybedtools.contrib.plotting.Track` class allows plotting of features with matplotlib. The `Track` class subclasses `matplotlib.collections.PolyCollection`, making it rather fast for 1000s of features.
- See the `scripts/pbt_plotting_example.py` script for a way of visually showing the results of BEDTools operations … great for teaching BEDTools to new users.
- New `BedTool.liftover()` method (needs a chain file from UCSC and the `liftover` program installed)
- `BedTool` creation using tuples/lists of values – everything is converted to string before creating an `Interval` object.
- bugfix: `BedTool.window_maker()` now handles the `genome` kwarg correctly
- bugfix: `pybedtools.cleanup(remove_all=True)` now works correctly when using the default temp dir

## Changes in v0.6

- Overhaul in online documentation to hopefully make functionality easier to find and/or discover. See pybedtools Reference for summary tables of the different parts of `pybedtools`; each entry is linked to further class/method/function-specific docs. These more detailed docs also have links to view the source code from within the HTML docs for more exploration.
- `pybedtools.contrib.venn_maker()` function that acts as an interface to the VennDiagram R package – just give it some BED files and it’ll do the rest.
- Debug mode – `pybedtools.debug_mode()` – for verbose logging messages.
- Fixed an open file leak (OSError: too many open files) that occured when opening thousands of streaming bed files in a single session.
- Initial support for tabix files. Useful for extracting features from a single region when you don’t need a full intersection.
- New `pybedtools.contrib` module (in the spirit of Django’s `contrib`) where higher-level functionality will be built.
- `pybedtools.contrib.Classifier` class for identifying the classes of intervals. Useful for making pie charts of intronic/exonic/intergenic etc classes of peaks. Note that this is somewhat redundant with the new `mapBed` program in BEDTools.
- Experimental `pybedtools.contrib.IntersectionMatrix` class for handling pairwise intersections of a large number of interval files – including a local sqlite3 database to avoid re-computing already up-to-date results.
- `Interval` objects are now hashable (it’s just a hash of the string representation) so that you can use them as dictionary keys.
- `BedTool.split()` method, which accepts a function returning an iterable of `Interval` objects. The function is applied to each interval. Useful for, say, splitting each gene into TSS, TTS, upstream and downstream features.
- `BedTool.truncate_to_chrom()` method, which truncates features to the chromosome sizes of the provided genome. Useful for when you try uploading a MACS-generated track to the UCSC genome browser, but it complains because peak boundaries have been extended outside chromosome boundaries … this method fixes the problem.
- `BedTool` objects now have full functionality of `IntervalFile` objects – that is, they have the methods `BedTool.any_hits()`, `BedTool.all_hits()`, and `BedTool.count_hits()` for doing single-interval tests. Sometimes this will be faster than using the tabix support, sometimes it won’t – it’s best to try both, depending on your data.
- String representations of `Interval` objects now have a newline at the end, just like a raw lines from a BED/GFF/VCF file. Previously, this was inconsistent and sometimes led to extra blank lines in “streaming” `BedTool` instances … which in turn led to problems with BEDTools programs using the chromsweep algorithm.
- Concatentate multiple files with one call to `BedTool.cat()` (thanks Jake Biesinger)
- Wrapped previous BEDTools programs:
    - `unionBedGraphs` (`BedTool.union_bedgraphs()`)
    - `pairToBed` (`BedTool.pair_to_bed()`)
    - `pairToPair` (`BedTool.pair_to_pair()`)
    - `bedpeToBam` (`BedTool.bedpe_to_bam()`)
- Wrapped new BEDTools programs:
    - `mapBed` (`BedTool.map()`)
    - `clusterBed` (`BedTool.cluster()`)
    - `randomBed` (`BedTool.random()`)
    - `multiIntersectBed` (`BedTool.multi_intersect()`)
    - `expandCols` (`BedTool.expand()`)
    - `windowMaker` (`BedTool.window_maker()`)
    - `bamToFastq` (`BedTool.bam_to_fastq()`)
- Made venn_gchart and venn_mpl tests more stable
- Automatic documenting of which args are passed implicitly for BedTool method calls
- More robust mechanisms for specifying custom paths for BEDTools installation as well as optional tabix, samtools, and R installations. This makes it easier to explicitly specify which versions of the tools to use.
- Improvements to GFF attributes: handle unescaped “=” (from sim4db GFFs) and make Attribute class properly dict-like (thanks Libor Mořkovský)

## Changes in v0.5.5

- Use `additional_args` kwarg to pass arguments verbatim to the underlying BEDTools programs. This is necessary for arguments like `genomeCoverageBed`’s `-5` argument, since `5=True` is not a valid Python expression. For example, you can use:

      import pybedtools
      a = pybedtools.example_bedtool('a.bed')
      a.genome_coverage(bg=True, strand='+', genome='hg19', additional_args='-5')

- Brent Pedersen added support for just 2 BED files in the Venn diagram scripts

- `BedTool.all_hits()` uses the underlying BEDTools C++ API to get all hits in a file for a particular Interval:

      a = pybedtools.example_bedtool('a.bed')
      interval = Interval('chr1', 1, 5000)
      a.all_hits(interval)

- New semantics for comparisons of Interval objects. Visual documentation of this coming soon.

- More tests for latest BEDTools code

- Interval instances are now pickleable; they can now be used across processes for parallel code.

## Changes in v0.5

- support for running random intersections in parallel. See `BedTool.randomstats()` and `BedTool.randomintersection()` (thanks, Jake Biesinger)
- Cython `Interval.__copy__()` for compatibility with `copy` module
- `seek()` and `rewind()` methods for `IntervalFile` class, used for Aaron Quinlan’s new chromsweep algorithm (https://github.com/arq5x/chrom_sweep) (thanks, Aaron)
- support and tests for new BEDTools programs `multiBamCov`, `tagBam`, and `nucBed`
- `output="out.bed"` kwarg for all wrapped methods for explicitly specifying where to save output – no more moving tempfiles
- docs improvements:
    - direct comparison with a shell script to illustrate benefit of `pybedtools`; see Shell script comparison
    - more installation details
    - 0- and 1-based coordinates discussed early on (the 3 brief examples page, Three brief examples)
    - development history and open collaboration model (see pybedtools development model)