    {-# OPTIONS_GHC -fno-warn-unused-imports #-}

    {-| @pipes-group@ builds upon @pipes@ to establish idioms for grouping streams
        into sub-streams without collecting elements into memory.  This tutorial
        assumes familiarity with @pipes@ and @pipes-parse@.
    -}

    module Pipes.Group.Tutorial (
        -- * Motivation
        -- $motivation

        -- * FreeT
        -- $freeT

        -- * How FreeT Works
        -- $advanced

        -- * Conclusion
        -- $conclusion
        ) where

    import Pipes
    import Pipes.Group

    {- $motivation
        Dividing a stream into sub-streams is non-trivial.  To illustrate the
        problem, consider the following task: limit a stream to the first three
        groups of elements (a group means consecutive equal elements).

        The wrong way to do it is to read each group into memory like this:

    > import Lens.Family.State.Strict (zoom)
    > import Pipes
    > import Pipes.Parse
    > import qualified Pipes.Prelude as P
    >
    > threeGroups :: (Monad m, Eq a) => Producer a m () -> Producer a m ()
    > threeGroups p0 = loop 3 p0
    >   where
    >     loop 0 _ = return ()
    >     loop n p = do
    >         (as, p') <- lift $ runStateT (zoom group drawAll) p
    >         each as
    >         loop (n - 1) p'

        The first problem is that this approach does not output any elements from
        each group until after parsing the entire group:

    >>> runEffect $ threeGroups P.stdinLn >-> P.stdoutLn
    1<Enter>
    1<Enter>
    2<Enter>
    1
    1
    2<Enter>
    2<Enter>
    3<Enter>
    2
    2
    2
    4<Enter>
    3
    >>>

        Worse, this program will crash without outputting a single value if fed an
        infinitely long group of identical elements:

    >>> runEffect $ threeGroups (each (repeat 1)) >-> P.print
    <Consumes all memory and crashes>

        A better approach is to just stream directly from the first three groups
        instead of storing the groups in intermediate lists:

    > import Lens.Family ((^.))
    > import Pipes
    > import Pipes.Parse
    > import qualified Pipes.Prelude as P
    >
    > threeGroups :: (Monad m, Eq a) => Producer a m () -> Producer a m ()
    > threeGroups p0 = loop 3 p0
    >   where
    >     loop 0 _ = return ()
    >     loop n p = do
    >         p' <- p ^. group
    >         loop (n - 1) p'

        This will run in constant memory and stream values immediately:

    >>> runEffect $ threeGroups P.stdinLn >-> P.stdoutLn
    1<Enter>
    1
    1<Enter>
    1
    2<Enter>
    2
    2<Enter>
    2
    2<Enter>
    2
    3<Enter>
    3
    4<Enter>

        However, this code is not very modular: we have to integrate our group
        creation logic with our group consumption logic.  This conflicts with the
        @pipes@ philosophy of decoupling streaming programs into modular components.

        An more modular approach would be to split our logic into three steps:

        * Split our 'Producer' into groups

        * Take the first three groups

        * Join these three groups back into a 'Producer'

        But how do we split our 'Producer' into groups without loading an entire
        group into memory?  We want to avoid solutions like the following code:

    > import Control.Monad (when, liftM2)
    > import Lens.Family.State.Strict (zoom)
    > import Pipes.Parse
    >
    > split :: (Monad m, Eq a) => Producer a m () -> Producer [a] m ()
    > split p = do
    >     ((as, eof), p') <- lift (runStateT parser p)
    >     yield as
    >     when (not eof) (split p')
    >   where
    >     parser = liftM2 (,) (zoom group drawAll) isEndOfInput

        ... because then we're back where we started, loading entire groups into
        memory.
    -}

    {- $freeT
        Fortunately, you can group elements while still streaming individual
        elements at a time.  The 'FreeT' type from the @free@ package solves this
        problem by allowing us to build \"linked lists\" of 'Producer's.  This lets
        you work with streams in a list-like manner.

        The key idea is that:

    > -- '~' means "is analogous to"
    >
    > -- If a Producer is like a list
    > Producer a m ()            ~   [a]
    >
    > -- ... then a 'FreeT'-delimited 'Producer' is like a list of lists
    > FreeT (Producer a m) m ()  ~  [[a]]

        Think of @(FreeT (Producer a m) m ())@ as a \"list of 'Producer's\".
        'FreeT' nests each subsequent 'Producer' within the return value of the
        previous 'Producer' so that you cannot access the next 'Producer' until you
        completely drain the current 'Producer'.  However, you rarely need to work
        with 'FreeT' directly.  Instead, you can structure most things using
        \"splitters\", \"transformations\" and \"joiners\":

    > -- A "splitter"
    > Producer a m ()           -> FreeT (Producer a m) m ()  ~   [a]  -> [[a]]
    >
    > -- A "transformation"
    > FreeT (Producer a m) m () -> FreeT (Producer a m) m ()  ~  [[a]] -> [[a]]
    >
    > -- A "joiner"
    > FreeT (Producer a m) m () -> Producer a m ()            ~  [[a]] ->  [a]

        An example splitter is @(view groups)@, which splits a 'Producer' into
        'FreeT'-delimited 'Producer's, one for each group of consecutive equal
        elements:

    > view groups :: (Eq a, Monad m) => Producer a m x -> FreeT (Producer a m) m x

        An example transformation is @(takes 3)@, which takes the first three
        'Producer's from a 'FreeT' and drops the rest:

    > takes 3 :: Monad m => FreeT (Producer a m) m () -> FreeT (Producer a m) m ()

        An example joiner is @concats@, which collapses a 'FreeT' of 'Producer's
        back down into a single 'Producer':

    > concats :: Monad m => FreeT (Producer a m) m x -> Producer a m x

        If you compose these three functions together, you will create a function
        that transforms a 'Producer' to keep only the first three groups of
        consecutive equal elements:

    > import Lens.Family
    > import Pipes
    > import Pipes.Group
    > import qualified Pipes.Prelude as P
    >
    > threeGroups :: (Monad m, Eq a) => Producer a m () -> Producer a m ()
    > threeGroups = concats . takes 3 . view groups

        Both splitting and joining preserve the streaming nature of 'Producer's and
        do not collect or buffer any values.  The transformed 'Producer' still
        outputs values immediately and does not wait for groups to complete before
        producing results.

    >>> runEffect $ threeGroups P.stdinLn >-> P.stdoutLn
    1<Enter>
    1
    1<Enter>
    1
    2<Enter>
    2
    2<Enter>
    2
    2<Enter>
    2
    3<Enter>
    3
    4<Enter>
    >>>

        Also, lenses simplify things even further.  The reason that 'groups' is a
        lens is because it actually combines both a splitter and joiner into a
        single package.  We can then use 'over' to handle both the splitting and
        joining for us:

    >>> runEffect $ over groups (takes 3) P.stdinLn >-> P.stdoutLn
    <Exact same behavior>

        This behaves the same because 'over' takes care of calling the splitter
        before applying the transformation, then calling the inverse joiner
        afterward.

        Another useful lens is 'individually', which lets you apply transformations
        to each 'Producer' layer of a 'FreeT'.  For example, if we wanted to
        add an extra @"!"@ line to the end of every group, we would write:

    >>> import Control.Applicative ((<*))
    >>> runEffect $ over (groups . individually) (<* yield "!") P.stdinLn >-> P.stdoutLn
    1<Enter>
    1
    1<Enter>
    1
    2<Enter>
    !
    2
    2<Enter>
    2
    2<Enter>
    2
    3<Enter>
    !
    3
    4<Enter>
    !
    >>>

        Note that 'individually' is only compatible with the @lens@ package.  You
        can alternatively use 'maps' if you are using @lens-family-core@:

    >>> runEffect $ over groups (maps (<* yield "!")) P.stdinLn >-> P.stdoutLn
    <Exact same behavior>

    -}

    {- $advanced
        You don't necessarily have to restrict yourself to predefined 'FreeT'
        functions.  You can also manually build or recurse over 'FreeT's of
        'Producer's.

        For example, here is how 'concats' is implemented, which collapses all the
        'Producer's within a 'FreeT' into a single 'Producer':

    > concats :: Monad m => FreeT (Producer a m) m x -> Producer a m x
    > concats = go
    >   where
    >     go f = do
    >         x <- lift (runFreeT f)  -- Match against the "head" of the "list"
    >         case x of
    >             Pure r -> return r  -- The "list" is empty
    >             Free p -> do        -- The "list" is non-empty
    >                 f' <- p         -- The return value of the 'Producer' is
    >                 go f'           --     the "tail" of the "list"

        Many patterns for 'FreeT's have equivalent analogs for lists.  'runFreeT'
        behaves like pattern matching on the list, except that you have to bind the
        result.  'Pure' is analogous to @[]@ and 'Free' is analogous to @(:)@.

        When you receive a 'Free' constructor that means you have a 'Producer' whose
        return value is the rest of the list (i.e. another 'FreeT').  You cannot
        access the rest of the list without running the 'Producer' to completion to
        retrieve this return value.  The above example just runs the entire
        'Producer', binds the remainder of the list to @f'@ and then recurses on
        that value.

        You can also build 'FreeT's in a manner similar to lists.  For example, the
        'chunksOf' lens uses the following splitter function internally:

    > _chunksOf :: Monad m => Producer a m x -> FreeT (Producer a m) m x
    > _chunksOf p = FreeT $ do
    >     x <- next p                     -- Pattern match on the 'Producer'
    >     return $ case x of
    >         Left   r      -> Pure r     -- Build an empty "list"
    >         Right (a, p') -> Free $ do  -- Build a non-empty "list"
    >             p'' <- (yield a >> p')^.splitAt n0  -- Emit the "head"
    >             return (_chunksOf p'')              -- Return the "tail"

        'Pure' signifies an empty 'FreeT' (one with no 'Producer' layers), just like
        @[]@ signifies an empty list (one with no elements).  We return 'Pure'
        whenever we cannot emit any more 'Producer's.

        'Free' indicates that we wish to emit a 'Producer' followed by another
        \"list\".  The 'Producer' we run directly within the body of the 'Free'.
        However, we store the remainder of the \"list\" within the return value of
        the 'Producer'.  This is where @_chunksOf@ recurses to build the rest of the
        \"list\".

        To gain a better understanding for how 'FreeT' works, consult the definition
        of the type, which you can find in "Control.Monad.Trans.Free":

    > newtype FreeT f m a = FreeT { runFreeT :: m (FreeF f a (FreeT f m a)) }
    >
    > data FreeF f a b = Pure a | Free (f b)

        ... and just replace all occurrences of @f@ with @(Producer e m)@:

    > -- This is pseudocode
    >
    > newtype FreeT' m a = FreeT { runFreeT :: m (FreeF' a (FreeT' m a)) }
    >
    > data FreeF' a b = Pure a | Free (Producer e m b)

        ... which you can further think of as:

    > -- More pseudocode
    >
    > newtype FreeT' m a =
    >     FreeT { runFreeT :: m (Pure a | Producer e m (FreeT' m a)) }

        In other words, 'runFreeT' unwraps a 'FreeT' to produce an action in the
        base monad which either finishes with a value of type @a@ or continues with
        a 'Producer' which returns a new 'FreeT'.  Vice versa, if you want to build
        a 'FreeT', you must create an action in the base monad which returns either
        a 'Pure' or a 'Producer' wrapping another 'FreeT'.
    -}

    {- $conclusion
        This library is very small since it only contains element-agnostic grouping
        utilities.  Downstream libraries that provide richer grouping utilities
        include @pipes-bytestring@ and @pipes-text@.

        To learn more about @pipes-group@, ask questions, or follow development, you
        can subscribe to the @haskell-pipes@ mailing list at:

        <https://groups.google.com/forum/#!forum/haskell-pipes>

        ... or you can mail the list directly at:

        <mailto:haskell-pipes@googlegroups.com>
    -}