git-annex/ profiling

This page exsists to collect profiling data about git-annex, so it can be referred to later. If you have a specific instance where git-annex seems unncessarily slow, please file a bug report about it.

comment 9

3x more allocations now than before. This is probably due to the switch to OsPath, which means extra copying from ByteString.

        Thu Aug 21 20:18 2025 Time and Allocation Profiling Report  (Final)

           git-annex +RTS -p -RTS find

        total time  =        0.99 secs   (989 ticks @ 1000 us, 1 processor)
        total alloc = 1,514,545,208 bytes  (excludes profiling overheads)

    COST CENTRE                        MODULE                         SRC                                                 %time %alloc

    keyFile                            Annex.Locations                Annex/Locations.hs:(790,1)-(806,44)                   7.1   12.9
    >>=.\                              Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:(148,9)-(149,44)    6.9    2.4
    >>=.\.succ'                        Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:148:13-76           5.6    0.7
    keyFile.esc                        Annex.Locations                Annex/Locations.hs:(796,9)-(801,32)                   5.3   10.7
    ifM                                Utility.Monad                  Utility/Monad.hs:(62,1)-(64,44)                       5.1    8.7
    ifM.\                              Utility.Monad                  Utility/Monad.hs:64:9-44                              3.5    8.6
    hashUpdates.processBlocks          Crypto.Hash                    Crypto/Hash.hs:(103,5)-(112,76)                       3.1    0.3
    inAnnex'.\                         Annex.Content.Presence         Annex/Content/Presence.hs:(53,61)-(68,31)             3.1    8.5
    keyFile.anyneedesc                 Annex.Locations                Annex/Locations.hs:806:9-44                           3.1    1.0
    seekFilteredKeys.exists            CmdLine.Seek                   CmdLine/Seek.hs:465:9-92                              2.8    0.4
    fileKey                            Annex.Locations                Annex/Locations.hs:(814,1)-(824,41)                   2.1    1.0
    keyPath                            Annex.Locations                Annex/Locations.hs:(834,1)-(836,23)                   1.8    5.9

Comment by joey — Wed Oct 29 14:05:27 2025
comment 8

Update after some recent optimisations involving seekFilteredKeys.

            Mon Oct 19 14:36 2020 Time and Allocation Profiling Report  (Final)

               git-annex +RTS -p -RTS find

            total time  =        1.32 secs   (1316 ticks @ 1000 us, 1 processor)
            total alloc = 583,359,800 bytes  (excludes profiling overheads)

    COST CENTRE           MODULE                              SRC                                                   %time %alloc

    keyFile               Annex.Locations                     Annex/Locations.hs:(602,1)-(612,30)                    13.2   32.7
    readObjectContent     Git.CatFile                         Git/CatFile.hs:(122,1)-(131,42)                        10.9    2.3
    copyAndFreeze         Data.ByteArray.Methods              Data/ByteArray/Methods.hs:(237,1)-(240,21)              5.2    0.8
    withPtr               Basement.Block.Base                 Basement/Block/Base.hs:(402,1)-(411,31)                 5.2    4.1
    parseLinkTarget       Annex.Link                          Annex/Link.hs:(263,1)-(271,25)                          4.5   12.3
    fileKey               Annex.Locations                     Annex/Locations.hs:(618,1)-(628,41)                     4.5    8.1
    decimal               Data.Attoparsec.ByteString.Char8    Data/Attoparsec/ByteString/Char8.hs:(447,1)-(448,49)    4.3    4.5
    doesPathExist         Utility.RawFilePath                 Utility/RawFilePath.hs:32:1-25                          3.9    0.6
    ifM                   Utility.Monad                       Utility/Monad.hs:(54,1)-(56,44)                         3.0    3.2
    catObjectStream       Git.CatFile                         Git/CatFile.hs:(322,1)-(330,45)                         2.6    2.5
    </>                   System.FilePath.Posix.ByteString    System/FilePath/Posix/../Internal.hs:841:1-15           2.4    3.4
    respParser            Git.CatFile                         Git/CatFile.hs:(175,1)-(182,39)                         2.4    1.3

Comment by joey — Wed Oct 29 14:05:27 2025
comment 7

Updated profiling. git-annex find is now ByteString end-to-end! Note the massive reduction in alloc, and improved runtime.

            Wed Dec 11 14:41 2019 Time and Allocation Profiling Report  (Final)

               git-annex +RTS -p -RTS find

            total time  =        1.51 secs   (1515 ticks @ 1000 us, 1 processor)
            total alloc = 608,475,328 bytes  (excludes profiling overheads)

    COST CENTRE                      MODULE                           SRC                                                    %time %alloc

    keyFile'                         Annex.Locations                  Annex/Locations.hs:(590,1)-(600,30)                      8.2   16.6
    >>=.\.succ'                      Data.Attoparsec.Internal.Types   Data/Attoparsec/Internal/Types.hs:146:13-76              4.7    0.7
    getAnnexLinkTarget'.probesymlink Annex.Link                       Annex/Link.hs:79:9-46                                    4.2    7.6
    >>=.\                            Data.Attoparsec.Internal.Types   Data/Attoparsec/Internal/Types.hs:(146,9)-(147,44)       3.9    2.3
    parseLinkTarget                  Annex.Link                       Annex/Link.hs:(255,1)-(263,25)                           3.9   11.8
    doesPathExist                    Utility.RawFilePath              Utility/RawFilePath.hs:30:1-25                           3.4    0.6
    keyFile'.esc                     Annex.Locations                  Annex/Locations.hs:(596,9)-(600,30)                      3.2   14.7
    fileKey'                         Annex.Locations                  Annex/Locations.hs:(609,1)-(619,41)                      3.0    4.7
    parseLinkTargetOrPointer         Annex.Link                       Annex/Link.hs:(240,1)-(244,25)                           2.8    0.2
    hashUpdates.\.\.\                Crypto.Hash                      Crypto/Hash.hs:85:48-99                                  2.5    0.1
    combineAlways                    System.FilePath.Posix.ByteString System/FilePath/Posix/../Internal.hs:(698,1)-(704,67)    2.0    3.3
    getState                         Annex                            Annex.hs:(251,1)-(254,27)                                2.0    1.1
    withPtr.makeTrampoline           Basement.Block.Base              Basement/Block/Base.hs:(401,5)-(404,31)                  1.9    1.7
    withMutablePtrHint               Basement.Block.Base              Basement/Block/Base.hs:(468,1)-(482,50)                  1.8    1.2
    parseKeyVariety                  Types.Key                        Types/Key.hs:(323,1)-(371,42)                            1.8    0.0
    fileKey'.go                      Annex.Locations                  Annex/Locations.hs:611:9-55                              1.7    2.2
    isLinkToAnnex                    Annex.Link                       Annex/Link.hs:(299,1)-(305,47)                           1.7    1.0
    hashDirMixed                     Annex.DirHashes                  Annex/DirHashes.hs:(82,1)-(90,27)                        1.7    1.3
    primitive                        Basement.Monad                   Basement/Monad.hs:72:5-18                                1.6    0.1
    withPtr                          Basement.Block.Base              Basement/Block/Base.hs:(395,1)-(404,31)                  1.5    1.6
    mkKeySerialization               Types.Key                        Types/Key.hs:(115,1)-(117,22)                            1.1    2.8
    decimal.step                     Data.Attoparsec.ByteString.Char8 Data/Attoparsec/ByteString/Char8.hs:448:9-49             0.8    1.2

Comment by joey — Wed Oct 29 14:05:27 2025
comment 6

After caching serialized Keys.

        Fri Nov 22 19:06 2019 Time and Allocation Profiling Report  (Final)

           git-annex +RTS -p -RTS find

        total time  =        3.00 secs   (2997 ticks @ 1000 us, 1 processor)
        total alloc = 1,890,060,432 bytes  (excludes profiling overheads)

    COST CENTRE                      MODULE                         SRC                                                 %time %alloc

    inAnnex'.\                       Annex.Content                  Annex/Content.hs:(103,61)-(118,31)                   28.2   40.6
    keyFile'                         Annex.Locations                Annex/Locations.hs:(564,1)-(574,30)                   3.8    5.3
    splitc                           Utility.Split                  Utility/Split.hs:(24,1)-(26,25)                       3.7    5.1
    _encodeFilePath                  Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:(111,1)-(114,49)        3.1    2.7
    encodeW8                         Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:(189,1)-(191,70)        3.0    3.6
    w82s                             Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:217:1-15                2.8    5.1
    >>=.\.succ'                      Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:146:13-76           2.8    0.2
    >>=.\                            Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:(146,9)-(147,44)    2.5    0.7
    getAnnexLinkTarget'.probesymlink Annex.Link                     Annex/Link.hs:78:9-62                                 2.4    2.5
    fileKey'                         Annex.Locations                Annex/Locations.hs:(583,1)-(593,41)                   2.1    1.5
    parseLinkTarget                  Annex.Link                     Annex/Link.hs:(254,1)-(262,25)                        2.0    3.8
    keyFile'.esc                     Annex.Locations                Annex/Locations.hs:(570,9)-(574,30)                   1.9    4.7
    s2w8                             Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:214:1-15                1.7    3.5
    keyPath                          Annex.Locations                Annex/Locations.hs:(603,1)-(605,23)                   1.6    3.5
    getState                         Annex                          Annex.hs:(251,1)-(254,27)                             1.6    0.3
    parseKeyVariety                  Types.Key                      Types/Key.hs:(322,1)-(370,42)                         1.4    0.0
    withMutablePtrHint               Basement.Block.Base            Basement/Block/Base.hs:(468,1)-(482,50)               1.3    0.4
    hashUpdates.\.\.\                Crypto.Hash                    Crypto/Hash.hs:85:48-99                               1.2    0.0
    parseLinkTargetOrPointer         Annex.Link                     Annex/Link.hs:(239,1)-(243,25)                        1.1    0.1
    primitive                        Basement.Monad                 Basement/Monad.hs:72:5-18                             1.1    0.0
    withPtr.makeTrampoline           Basement.Block.Base            Basement/Block/Base.hs:(401,5)-(404,31)               1.0    0.6
    assertLocal                      Git                            Git.hs:(123,1)-(129,28)                               0.8    1.6
    decodeBS'                        Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:151:1-31                0.4    2.4

Runtime improved by 5% or so, and getAnnexLinkTarget moved up, otherwise not a lot of change. keyFile is looking like an optimization target, although its percent of the runtime actually reduced. However that's specific to this repo which has a lot of URL keys that contain '/' and so need to be escaped.

        Fri Nov 22 19:09 2019 Time and Allocation Profiling Report  (Final)

           git-annex +RTS -p -RTS find --not --in web

        total time  =        8.42 secs   (8421 ticks @ 1000 us, 1 processor)
        total alloc = 1,887,547,744 bytes  (excludes profiling overheads)

    COST CENTRE                      MODULE                           SRC                                                 %time %alloc

    catObjectDetails.\               Git.CatFile                      Git/CatFile.hs:(83,88)-(91,97)                        8.1    4.1
    catchMaybeIO                     Utility.Exception                Utility/Exception.hs:53:1-63                          7.6    2.2
    parseResp                        Git.CatFile                      Git/CatFile.hs:(145,1)-(156,28)                       5.2    5.6
    >>=.\                            Data.Attoparsec.Internal.Types   Data/Attoparsec/Internal/Types.hs:(146,9)-(147,44)    5.0    2.0
    >>=.\.succ'                      Data.Attoparsec.Internal.Types   Data/Attoparsec/Internal/Types.hs:146:13-76           4.7    1.0
    MAIN                             MAIN                             <built-in>                                            4.2    0.4
    getState                         Annex                            Annex.hs:(251,1)-(254,27)                             2.0    1.1
    getAnnexLinkTarget'.probesymlink Annex.Link                       Annex/Link.hs:78:9-62                                 1.9    2.6
    splitc                           Utility.Split                    Utility/Split.hs:(24,1)-(26,25)                       1.9    5.2
    _encodeFilePath                  Utility.FileSystemEncoding       Utility/FileSystemEncoding.hs:(111,1)-(114,49)        1.8    2.8
    query.send                       Git.CatFile                      Git/CatFile.hs:141:9-32                               1.7    0.5
    keyFile'                         Annex.Locations                  Annex/Locations.hs:(564,1)-(574,30)                   1.6    5.4

Ditto.

Comment by joey — Wed Oct 29 14:05:27 2025
comment 5

After switching many internal types to ByteString.

(Note that stack build --profile built this with -O, not -O2, so it's not as fast as it ought to be, but the cost centers are probably fairly accurate still.)

            Mon Jan 14 17:17 2019 Time and Allocation Profiling Report  (Final)

               git-annex +RTS -p -RTS find

            total time  =        3.07 secs   (3074 ticks @ 1000 us, 1 processor)
            total alloc = 1,880,855,184 bytes  (excludes profiling overheads)

    COST CENTRE                      MODULE                         SRC                                                 %time %alloc

    inAnnex'.checkindirect           Annex.Content                  Annex/Content.hs:(106,9)-(121,39)                    31.6   41.0
    splitc                           Utility.Split                  Utility/Split.hs:(24,1)-(26,25)                       4.8    5.2
    keyFile'                         Annex.Locations                Annex/Locations.hs:(518,1)-(524,30)                   4.5    5.2
    encodeW8                         Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:(189,1)-(191,70)        3.1    3.6
    >>=.\                            Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:(146,9)-(147,44)    2.5    0.7
    _encodeFilePath                  Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:(111,1)-(114,49)        2.5    2.7
    >>=.\.succ'                      Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:146:13-76           2.2    0.2
    fileKey'                         Annex.Locations                Annex/Locations.hs:(532,1)-(541,41)                   2.2    1.5
    getState                         Annex                          Annex.hs:(254,1)-(257,27)                             2.1    0.4
    getAnnexLinkTarget'.probesymlink Annex.Link                     Annex/Link.hs:77:9-62                                 1.9    2.5
    w82s                             Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:217:1-15                1.8    5.2
    keyPath                          Annex.Locations                Annex/Locations.hs:(551,1)-(553,23)                   1.7    3.5
    keyFile'.esc                     Annex.Locations                Annex/Locations.hs:(520,9)-(524,30)                   1.6    4.7
    fileKey'.go                      Annex.Locations                Annex/Locations.hs:535:9-63                           1.6    1.6
    s2w8                             Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:214:1-15                1.3    3.5
    withPtr                          Basement.Block.Base            Basement/Block/Base.hs:(395,1)-(404,31)               1.3    0.5
    parseLinkTarget                  Annex.Link                     Annex/Link.hs:(247,1)-(255,25)                        1.2    3.8
    parseKeyVariety                  Types.Key                      Types/Key.hs:(135,1)-(184,41)                         1.2    0.0
    assertLocal                      Git                            Git.hs:(123,1)-(129,28)                               0.8    1.6
    decodeBS'                        Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:151:1-31                0.6    2.4

Notice that the percent of time inAnnex' went up from 14.1% to 31.6%. That and getAnnexLinkTarget are the meat of the IO, so it's good for them to get a higher percent of the CPU, to the extent they're IO bound. It seems like getAnnexLinkTarget also lost a lot of non-IO overhead.

There are still some overheads from conversion to and from ByteString, but the above does seem like a good improvement.

            Mon Jan 14 17:56 2019 Time and Allocation Profiling Report  (Final)

               git-annex +RTS -p -RTS find --not --in web

            total time  =        7.62 secs   (7622 ticks @ 1000 us, 1 processor)
            total alloc = 1,908,064,368 bytes  (excludes profiling overheads)

    COST CENTRE                      MODULE                         SRC                                                 %time %alloc

    catObjectDetails.\               Git.CatFile                    Git/CatFile.hs:(83,88)-(91,97)                        6.5    3.8
    catchDefaultIO                   Utility.Exception              Utility/Exception.hs:57:1-53                          6.4    2.5
    parseResp                        Git.CatFile                    Git/CatFile.hs:(141,1)-(152,28)                       4.9    5.0
    MAIN                             MAIN                           <built-in>                                            4.6    0.4
    >>=.\                            Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:(146,9)-(147,44)    4.6    1.7
    >>=.\.succ'                      Data.Attoparsec.Internal.Types Data/Attoparsec/Internal/Types.hs:146:13-76           4.1    0.7
    getState                         Annex                          Annex.hs:(254,1)-(257,27)                             2.7    1.1
    simplifyPath                     Utility.Path                   Utility/Path.hs:(38,1)-(50,48)                        2.6    6.8
    splitc                           Utility.Split                  Utility/Split.hs:(24,1)-(26,25)                       2.5    5.1
    keyFile'                         Annex.Locations                Annex/Locations.hs:(518,1)-(524,30)                   1.7    5.2
    getAnnexLinkTarget'.probesymlink Annex.Link                     Annex/Link.hs:77:9-62                                 1.7    2.6
    journalFile                      Annex.Journal                  Annex/Journal.hs:(107,1)-(112,33)                     1.7    5.7
    catches                          Control.Monad.Catch            src/Control/Monad/Catch.hs:(795,1)-(799,76)           1.7    2.7
    query.send                       Git.CatFile                    Git/CatFile.hs:137:9-32                               1.6    0.5
    delEntry                         Utility.Env                    Utility/Env.hs:(57,1)-(60,48)                         1.6    0.8
    encodeW8                         Utility.FileSystemEncoding     Utility/FileSystemEncoding.hs:(189,1)-(191,70)        1.4    3.5
    query                            Git.CatFile                    Git/CatFile.hs:(130,1)-(138,26)                       1.3    0.0

Notice that allocations dropped by 1/3rd!

Otherwise, not a large change here..

Comment by joey — Wed Oct 29 14:05:27 2025
comment 11

Switched from MissingH to cryptonite for md5. It did move md5 out of the top CPU spot but the overall runtime didn't change much. Memory allocations did go down by a good amount.

Updated profiles:

               git-annex +RTS -p -RTS find

            total time  =        1.63 secs   (1629 ticks @ 1000 us, 1 processor)
            total alloc = 1,496,336,496 bytes  (excludes profiling overheads)

    COST CENTRE              MODULE                     SRC                                             %time %alloc

    catchIO                  Utility.Exception          Utility/Exception.hs:79:1-17                     14.1   15.1
    inAnnex'.checkindirect   Annex.Content              Annex/Content.hs:(108,9)-(119,39)                10.6    4.8
    catches                  Control.Monad.Catch        src/Control/Monad/Catch.hs:(432,1)-(436,76)       8.6    6.9
    spanList                 Data.List.Utils            src/Data/List/Utils.hs:(150,1)-(155,36)           6.7   11.1
    isAnnexLink              Annex.Link                 Annex/Link.hs:35:1-85                             5.0   10.2
    keyFile                  Annex.Locations            Annex/Locations.hs:(456,1)-(462,19)               5.0    7.0
    readish                  Utility.PartialPrelude     Utility/PartialPrelude.hs:(48,1)-(50,20)          3.8    2.0
    startswith               Data.List.Utils            src/Data/List/Utils.hs:103:1-23                   3.6    2.3
    splitc                   Utility.Misc               Utility/Misc.hs:(52,1)-(54,25)                    3.4    6.5
    s2w8                     Data.Bits.Utils            src/Data/Bits/Utils.hs:65:1-15                    2.6    6.4
    keyPath                  Annex.Locations            Annex/Locations.hs:(492,1)-(494,23)               2.5    4.4
    fileKey.unesc            Annex.Locations            Annex/Locations.hs:(469,9)-(474,39)               2.0    3.5
    copyAndFreeze            Data.ByteArray.Methods     Data/ByteArray/Methods.hs:(224,1)-(227,21)        1.8    0.5

               git-annex +RTS -p -RTS find --not --in web

            total time  =        5.33 secs   (5327 ticks @ 1000 us, 1 processor)
            total alloc = 2,908,489,000 bytes  (excludes profiling overheads)

    COST CENTRE          MODULE                     SRC                                             %time %alloc

    catObjectDetails.\   Git.CatFile                Git/CatFile.hs:(80,72)-(88,97)                    7.8    2.8
    catchIO              Utility.Exception          Utility/Exception.hs:79:1-17                      7.6    8.3
    spanList             Data.List.Utils            src/Data/List/Utils.hs:(150,1)-(155,36)           5.8    9.1
    readish              Utility.PartialPrelude     Utility/PartialPrelude.hs:(48,1)-(50,20)          4.5    4.0
    parseResp            Git.CatFile                Git/CatFile.hs:(113,1)-(124,28)                   4.4    2.9
    readFileStrict       Utility.Misc               Utility/Misc.hs:33:1-59                           3.7    1.6
    catches              Control.Monad.Catch        src/Control/Monad/Catch.hs:(432,1)-(436,76)       3.1    3.6
    encodeW8             Utility.FileSystemEncoding Utility/FileSystemEncoding.hs:(131,1)-(133,70)    3.1    2.3

Comment by joey — Wed Oct 29 14:05:27 2025
comment 10

- Optimised key2file and file2key. 18% scanning time speedup.
- Optimised adjustGitEnv. 50% git-annex branch query speedup
- Optimised parsePOSIXTime. 10% git-annex branch query speedup
- Tried making catObjectDetails.receive use ByteString for parsing, but that did not seem to speed it up significantly. So it parsing is already fairly optimal, it's just that a lot of data passes through it when querying the git-annex branch.

After all that, profiling `git-annex find`:

            Thu Sep 29 16:51 2016 Time and Allocation Profiling Report  (Final)

               git-annex.1 +RTS -p -RTS find

            total time  =        1.73 secs   (1730 ticks @ 1000 us, 1 processor)
            total alloc = 1,812,406,632 bytes  (excludes profiling overheads)

    COST CENTRE            MODULE                  %time %alloc

    md5                    Data.Hash.MD5            28.0   37.9
    catchIO                Utility.Exception        10.2   12.5
    inAnnex'.checkindirect Annex.Content             9.9    3.7
    catches                Control.Monad.Catch       8.7    5.7
    readish                Utility.PartialPrelude    5.7    3.0
    isAnnexLink            Annex.Link                5.0    8.4
    keyFile                Annex.Locations           4.2    5.8
    spanList               Data.List.Utils           4.0    6.3
    startswith             Data.List.Utils           2.0    1.3

And `git-annex find --not --in web`:

            Thu Sep 29 16:35 2016 Time and Allocation Profiling Report  (Final)

               git-annex +RTS -p -RTS find --not --in web

            total time  =        5.24 secs   (5238 ticks @ 1000 us, 1 processor)
            total alloc = 3,293,314,472 bytes  (excludes profiling overheads)

    COST CENTRE               MODULE                      %time %alloc

    catObjectDetails.receive  Git.CatFile                  12.9    5.5
    md5                       Data.Hash.MD5                10.6   20.8
    readish                   Utility.PartialPrelude        7.3    8.2
    catchIO                   Utility.Exception             6.7    7.3
    spanList                  Data.List.Utils               4.1    7.4
    readFileStrictAnyEncoding Utility.Misc                  3.5    1.3
    catches                   Control.Monad.Catch           3.3    3.2

So, quite a large speedup overall!

This leaves md5 still unoptimised at 10-28% of CPU use. I looked at switching it to cryptohash's implementation, but it would require quite a lot of bit-banging math to pull the used values out of the ByteString containing the md5sum.

Comment by joey — Wed Oct 29 14:05:27 2025
more profiling

Instead of profiling `git annex copy --to remote`, I profiled `git annex find --not --in web`, which needs to do the same kind of location log lookup.

            total time  =       12.41 secs   (12413 ticks @ 1000 us, 1 processor)
            total alloc = 8,645,057,104 bytes  (excludes profiling overheads)

    COST CENTRE               MODULE                      %time %alloc

    adjustGitEnv              Git.Env                      21.4   37.0
    catchIO                   Utility.Exception            13.2    2.8
    spanList                  Data.List.Utils              12.6   17.9
    parsePOSIXTime            Logs.TimeStamp                6.1    5.0
    catObjectDetails.receive  Git.CatFile                   5.9    2.1
    startswith                Data.List.Utils               5.7    3.8
    md5                       Data.Hash.MD5                 5.1    7.9
    join                      Data.List.Utils               2.4    6.0
    readFileStrictAnyEncoding Utility.Misc                  2.2    0.5

The adjustGitEnv overhead is a surprise! It seems it is getting called once per file, and allocating a new copy of the environment each time. Call stack: withIndex calls withIndexFile calls addGitEnv calls adjustGitEnv. Looks like simply making gitEnv be cached at startup would avoid most of the adjustGitEnv slowdown.

(The catchIO overhead is a false reading; the detailed profile shows that all its time and allocations are inherited. getAnnexLinkTarget is running catchIO in the expensive case, so readSymbolicLink is the actual expensive bit.)

The parsePOSIXTime comes from reading location logs. It's implemented using a generic Data.Time.Format.parseTime, which uses a format string "%s%Qs". A custom parser that splits into seconds and picoseconds and simply reads both numbers might be more efficient.

catObjectDetails.receive is implemented using mostly String and could probably be sped up by being converted to use ByteString.

Comment by joey — Wed Oct 29 14:05:27 2025
profiling

Built git-annex with profiling, using `stack build --profile`

(For reproduciblity, running git-annex in a clone of the git-annex repo https://github.com/RichiH/conference_proceedings with rev 2797a49023fc24aff6fcaec55421572e1eddcfa2 checked out. It has 9496 annexed objects.)

Profiling `git-annex find +RTS -p`:

            total time  =        3.53 secs   (3530 ticks @ 1000 us, 1 processor)
            total alloc = 3,772,700,720 bytes  (excludes profiling overheads)

    COST CENTRE            MODULE                  %time %alloc

    spanList               Data.List.Utils          32.6   37.7
    startswith             Data.List.Utils          14.3    8.1
    md5                    Data.Hash.MD5            12.4   18.2
    join                   Data.List.Utils           6.9   13.7
    catchIO                Utility.Exception         5.9    6.0
    catches                Control.Monad.Catch       5.0    2.8
    inAnnex'.checkindirect Annex.Content             4.6    1.8
    readish                Utility.PartialPrelude    3.0    1.4
    isAnnexLink            Annex.Link                2.6    4.0
    split                  Data.List.Utils           1.5    0.8
    keyPath                Annex.Locations           1.2    1.7

This is interesting!

Fully 40% of CPU time and allocations are in list (really String) processing, and the details of the profiling report show that `spanList` and `startsWith` and `join` are all coming from calls to `replace` in `keyFile` and `fileKey`. Both functions nest several calls to replace, so perhaps that could be unwound into a single pass and/or a ByteString used to do it more efficiently.

12% of run time is spent calculating the md5 hashes for the hash directories for .git/annex/objects. Data.Hash.MD5 is from missingh, and it is probably a quite unoptimised version. Switching to the version if cryptonite would probably speed it up a lot.

Comment by joey — Wed Oct 29 14:05:27 2025
Comments on this page are closed.