a ,_@sLdZddlmZddlZddlmZmZddlmZddlZddl Z ddl m Z ddl Z ddl Z ddlmZddl mZmZdd l mZddlZz ddlZWneyd ZYn0d Ze jd krd Zdd lmZmZddlmZddlmZddlm Z m!Z!m"Z"m#Z#dZ$dZ%e$e%fZ&dZ'doddZ(Gddde)Z*Gddde+Z,dZ-d e-fddZ.ddZ/dd Z0dpd!d"Z1d#d$Z2d%d&Z3Gd'd(d(e+Z4Gd)d*d*e4Z5Gd+d,d,e4Z6Gd-d.d.e6Z7Gd/d0d0e6Z8d1d2Z9d3d4Z:dd d e-fd5d6Z;d7d8Zd=d>Z?dqd@dAZ@GdBdCdCe+ZAGdDdEdEe+ZBGdFdGdGeBZCGdHdIdIeBZDGdJdKdKe+ZEGdLdMdMe+ZFdNdOZGdrdQdRZHdsdSdTZIdUdVZJdtdWdXZKdYdZZLdud[d\ZMdvd]d^ZNd_d`ZOdadbZPdcZQdddeZRdfdgZSdhdiZTdjdkZUeUZVGdldmdme+ZWzddnlXmTZTm3Z3WneyFYn0dS)waClasses for dealing with packed git objects. A pack is a compact representation of a bunch of objects, stored using deltas where possible. They have two parts, the pack file, which stores the data, and an index that tells you where the data is. To find an object you look in all of the index files 'til you find a match for the object name. You then use the pointer got from this as a pointer in to the corresponding packfile. ) defaultdictN)BytesIOUnsupportedOperation)deque)chain)sha1)SEEK_CURSEEK_END) unpack_fromFTZPlan9)ApplyDeltaErrorChecksumMismatch)GitFile) LRUSizeCache)ShaFile hex_to_sha sha_to_hex object_header cCsXg}t|dks|dd@rP|d}|dur8t||}|t|ddq||fS)zVRead bytes marked with most significant bit. Args: read: Read function rN)lenbinasciicrc32appendord)readrretbr!./usr/lib/python3/dist-packages/dulwich/pack.pytake_msb_bytes^s r#c@seZdZddZdS)PackFileDisappearedcCs ||_dSN)objselfr&r!r!r"__init__oszPackFileDisappeared.__init__N)__name__ __module__ __qualname__r)r!r!r!r"r$msr$c@sPeZdZdZgdZddZddZddZd d Zd d Z d dZ ddZ dS)UnpackedObjectaClass encapsulating an object unpacked from a pack file. These objects should only be created from within unpack_object. Most members start out as empty and are filled in at various points by read_zlib_chunks, unpack_object, DeltaChainIterator, etc. End users of this object should take care that the function they're getting this object from is guaranteed to set the members they need. ) offset_sha obj_type_num obj_chunks pack_type_num delta_base comp_chunks decomp_chunks decomp_lenrcCs^d|_d|_||_||_d|_g|_||_||_|tvrFd|_ d|_ n||_ |j|_ ||_dSr%) r.r/r2r3r4r5r6r DELTA_TYPESr0r1)r(r2r3r6rr!r!r"r)szUnpackedObject.__init__cCs |jdurt|j|j|_|jS)z%Return the binary SHA of this object.N)r/obj_shar0r1r(r!r!r"shas zUnpackedObject.shacCst|j|jS)z"Return a ShaFile from this object.)rZfrom_raw_chunksr0r1r9r!r!r"sha_fileszUnpackedObject.sha_filecCs |jtvr|j|jfS|jSdS)z>Return the decompressed chunks, or (delta base, delta chunks).N)r2r7r3r5r9r!r!r"_objs  zUnpackedObject._objcCs8t|tsdS|jD]}t||t||krdSqdSNFT) isinstancer- __slots__getattr)r(otherZslotr!r!r"__eq__s   zUnpackedObject.__eq__cCs ||k Sr%r!r(rAr!r!r"__ne__szUnpackedObject.__ne__cs*fddjD}djjd|fS)Ncsg|]}d|t|fqS)z%s=%r)r@.0sr9r!r" z+UnpackedObject.__repr__..z%s(%s)z, )r? __class__r*joinr(datar!r9r"__repr__szUnpackedObject.__repr__N) r*r+r,__doc__r?r)r:r;r<rBrDrNr!r!r!r"r-ss r-ic Cs|jdkrtdt}g}|j}d}|j}||} | sDtd|| || } |t | 7}|| |j } | rt | } |durt | d| |}|r| d| |d<qq.|dur.t | |}q.|dur|dM}||jkrtd||_|r||_ | S)aRead zlib data from a buffer. This function requires that the buffer have additional data following the compressed data, which is guaranteed to be the case for git pack files. Args: read_some: Read function that returns at least one byte, but may return less than the requested size. unpacked: An UnpackedObject to write result data to. If its crc32 attr is not None, the CRC32 of the compressed bytes will be computed using this starting CRC32. After this function, will have the following attrs set: * comp_chunks (if include_comp is True) * decomp_chunks * decomp_len * crc32 include_comp: If True, include compressed data in the result. buffer_size: Size of the read buffer. Returns: Leftover unused data from the decompression. Raises: zlib.error: if a decompression error occurred. rz+non-negative zlib data stream size expectedrzEOF before end of zlib streamNz.decompressed data does not match expected size) r6 ValueErrorzlibZ decompressobjr5rerrorr decompressrZ unused_datarr4) read_someunpacked include_comp buffer_sizeZ decomp_objr4r5r6raddZdecompunusedleftr!r!r"read_zlib_chunkss@        r\cCs(t}|D]}||q |dS)zReturn the hexdigest of the SHA1 over a set of names. Args: iter: Iterator over string objects Returns: 40-byte hex sha1 digest ascii)rupdate hexdigestencode)iterr:namer!r!r" iter_sha1s rccCs8t|d}t||WdS1s*0YdS)zLoad an index file by path. Args: filename: Path to the index file Returns: A PackIndex loaded from the given path rbN)r load_pack_index_file)pathfr!r!r"load_pack_indexs rhc Csz |}Wnttfy&d}Yn0|dur||durDt|j}tr|ztj||tjd}Wntj yrYn 0||fS| }t |}||fS)N)access) filenorAttributeErrorosfstatst_sizehas_mmapmmapZ ACCESS_READrSrr)rgsizefdcontentsr!r!r"_load_file_contentss    rtcCsnt|\}}|dddkrZtd|ddd}|dkrLt||||dStd |nt||||dSdS) zLoad an index file from a file-like object. Args: path: Path for the index file f: File-like object Returns: A PackIndex loaded from the given file NtOc>Lr)filersrqzUnknown pack index format %d)rtstructunpack PackIndex2KeyError PackIndex1)rfrgrsrqversionr!r!r"re2s recCsV||ks J||krR||d}||}||kr:|d}q ||krL|d}q |Sq dS)aFind a SHA in a data blob with sorted SHAs. Args: start: Start index of range to search end: End index of range to search sha: Sha to find unpack_name: Callback to retrieve SHA by index Returns: Index of the SHA, or None if it wasn't found ryrNr!)startendr:Z unpack_nameiZfile_shar!r!r"bisect_find_shaFs    rc@sheZdZdZddZddZddZdd Zd d Zd d Z ddZ ddZ ddZ ddZ ddZdS) PackIndexzAn index in to a packfile. Given a sha id of an object a pack index can tell you the location in the packfile of that object if it has it. cCsHt|tsdSt||D]"\\}}}\}}}||kr dSq dSr=)r>rzip iterentries)r(rAZname1_Zname2r!r!r"rBds zPackIndex.__eq__cCs || Sr%)rBrCr!r!r"rDnszPackIndex.__ne__cCst|jdS)0Return the number of entries in this pack index.N)NotImplementedError__len__r9r!r!r"rqszPackIndex.__len__cCstt|S)z#Iterate over the SHAs in this pack.)mapr_itershar9r!r!r"__iter__uszPackIndex.__iter__cCst|jdSzIterate over the entries in this pack index. Returns: iterator over tuples with object name, offset in packfile and crc32 checksum. N)rrr9r!r!r"ryszPackIndex.iterentriescCst|jdS)pReturn the SHA1 checksum stored for the corresponding packfile. Returns: 20-byte binary digest N)rget_pack_checksumr9r!r!r"rszPackIndex.get_pack_checksumcCsXt|dkrt|}z ||WStyRt|jdd}|dvrLt|Yn0dS)aReturn the index in to the corresponding packfile for the object. Given the name of an object it will return the offset that object lives at within the corresponding pack file. If the pack file doesn't have the object then None will be returned. (closedN)NT)rr _object_indexrQr@ _contentsr$)r(r:rr!r!r" object_indexs   zPackIndex.object_indexcCs0|D]\}}}||kr|Sqt|dS)zEReturn the SHA1 corresponding to the index in the pack file. N)rr~)r(indexrbr.rr!r!r" object_sha1s zPackIndex.object_sha1cCst|jdS)eSee object_index. Args: sha: A *binary* SHA string. (20 characters long)_ N)rrr(r:r!r!r"rszPackIndex._object_indexcCs t|S)zReturn the hex SHA1 over all the shas of all objects in this pack. Note: This is used for the filename of the pack. )rcrr9r!r!r" objects_sha1szPackIndex.objects_sha1cCst|jdS)z9Yield all the SHA1's of the objects in the index, sorted.N)rrr9r!r!r"rszPackIndex._itershaN)r*r+r,rOrBrDrrrrrrrrrr!r!r!r"r]s  rc@sJeZdZdZdddZddZddZd d Zd d Zd dZ ddZ dS)MemoryPackIndexz-Pack index that is stored entirely in memory.NcCs@i|_i|_|D]\}}}||j|<||j|<q||_||_dS)zCreate a new MemoryPackIndex. Args: entries: Sequence of name, idx, crc32 (sorted) pack_checksum: Optional pack checksum N)_by_sha _by_index_entries_pack_checksum)r(entries pack_checksumrbidxrr!r!r"r)s  zMemoryPackIndex.__init__cCs|jSr%)rr9r!r!r"rsz!MemoryPackIndex.get_pack_checksumcCs t|jSr%)rrr9r!r!r"rszMemoryPackIndex.__len__cCs|j|dSNr)rrr!r!r"rszMemoryPackIndex._object_indexcCs |j|Sr%)r)r(rr!r!r"rszMemoryPackIndex.object_sha1cCs t|jSr%)rarr9r!r!r"rszMemoryPackIndex._itershacCs t|jSr%)rarr9r!r!r"rszMemoryPackIndex.iterentries)N) r*r+r,rOr)rrrrrrr!r!r!r"rs rcseZdZdZd%ddZeddZfddZd d Zd d Z d dZ ddZ ddZ ddZ ddZddZddZddZddZdd Zd!d"Zd#d$ZZS)& FilePackIndexaPack index that is based on a file. To do the loop it opens the file, and indexes first 256 4 byte groups with the first byte of the sha id. The value in the four byte group indexed is the end of the group that shares the same starting byte. Subtract one from the starting byte and index again to find the start of the group. The values are sorted by sha id within the group, so do the math to find the start and end offset and then bisect in to find if the value is present. NcCsR||_|durt|d|_n||_|dur@t|j|\|_|_n|||_|_dS)zCreate a pack index object. Provide it with the name of the index file to consider, and it will map it whenever required. Nrd) _filenamer _filertr_sizer(filenamerzrsrqr!r!r"r)szFilePackIndex.__init__cCs|jSr%rr9r!r!r"rfszFilePackIndex.pathcs*t|tr|j|jkrdStt||S)NF)r>r_fan_out_tablesuperrBrCrJr!r"rBs   zFilePackIndex.__eq__cCs*|jt|jdddur&|jdS)Nclose)rrr@rr9r!r!r"rs zFilePackIndex.closecCs |jdS)rr)rr9r!r!r"r szFilePackIndex.__len__cCst|jdS)zUnpack the i-th entry in the index file. Returns: Tuple with object name (SHA), offset in pack file and CRC32 checksum (if known). N)r _unpack_entryr(rr!r!r"rszFilePackIndex._unpack_entrycCst|jdS)z)Unpack the i-th name from the index file.N)r _unpack_namerr!r!r"rszFilePackIndex._unpack_namecCst|jdS)z2Unpack the i-th object offset from the index file.N)r_unpack_offsetrr!r!r"rszFilePackIndex._unpack_offsetcCst|jdS)zJUnpack the crc32 checksum for the ith object from the index file. N)r_unpack_crc32_checksumrr!r!r"rsz$FilePackIndex._unpack_crc32_checksumccs"tt|D]}||Vq dSr%)rangerrrr!r!r"r$szFilePackIndex._itershaccs"tt|D]}||Vq dSr)rrrrr!r!r"r(szFilePackIndex.iterentriescCsNg}tdD]<}|j||d||dd}|td|dq |S)Nrur>Lr)rrrr{r|)r(Z start_offsetrrZ fanout_entryr!r!r"_read_fan_out_table1s z!FilePackIndex._read_fan_out_tablecCs&|}|}||kr"t||dS)z;Check that the stored checksum matches the actual checksum.Ncalculate_checksumget_stored_checksumr r(Zactualstoredr!r!r"check9szFilePackIndex.checkcCst|jddS)zlCalculate the SHA1 checksum over this pack index. Returns: This is a 20-byte binary digest N)rrdigestr9r!r!r"r@sz FilePackIndex.calculate_checksumcCst|jddS)rirbytesrr9r!r!r"rGszFilePackIndex.get_pack_checksumcCst|jddS)z`Return the SHA1 checksum stored for this index. Returns: 20-byte binary digest rNrr9r!r!r"rNsz!FilePackIndex.get_stored_checksumcCspt|dksJt|dd}|dkr.d}n|j|d}|j|}t||||j}|durft|||S)rNrr)rrrrrr~r)r(r:rrrrr!r!r"rUs zFilePackIndex._object_index)NNN)r*r+r,rOr)propertyrfrBrrrrrrrrrrrrrr __classcell__r!r!rr"rs&    rcsBeZdZdZd fdd ZddZddZd d Zd d ZZ S)rzVersion 1 Pack Index file.Ncs,tt|||||d|_|d|_dS)Nrr)rrr)rrrrrr!r"r)kszPackIndex1.__init__cCs$td|jd|d\}}||dfS)N>L20sr r)r(rr.rbr!r!r"rps zPackIndex1._unpack_entrycCs"d|dd}|j||dS)Nrrrur)rr(rr.r!r!r"ruszPackIndex1._unpack_namecCsd|d}td|j|dS)Nrrrrrrr!r!r"rys zPackIndex1._unpack_offsetcCsdSr%r!rr!r!r"r}sz!PackIndex1._unpack_crc32_checksum)NNN r*r+r,rOr)rrrrrr!r!rr"rhs rcsBeZdZdZd fdd ZddZddZd d Zd d ZZ S)r}zVersion 2 Pack Index file.Ncstt||||||jdddkr0tdtd|jd\|_|jdkrZtd|j|d|_d|_ |j d t ||_ |j dt ||_ |j dt ||_ dS) NrurvzNot a v2 pack index filerwryVersion was %drxir)rr}r)rAssertionErrorr rrr_name_table_offsetr_crc32_table_offset_pack_offset_table_offset_pack_offset_largetable_offsetrrr!r"r)s   zPackIndex2.__init__cCs||||||fSr%)rrrrr!r!r"rszPackIndex2._unpack_entrycCs |j|d}|j||dSNr)rrrr!r!r"rszPackIndex2._unpack_namecCsP|j|d}td|j|d}|d@rL|j|d@d}td|j|d}|S)Nrurrirxz>Q)rr rrrr!r!r"rs zPackIndex2._unpack_offsetcCstd|j|j|ddS)Nrrur)r rrrr!r!r"rs  z!PackIndex2._unpack_crc32_checksum)NNNrr!r!rr"r}s  r}cCsd|d}|sdS|dddkr,td|td|d\}|dvrNtd |td|d \}||fS) zRead the header of a pack file. Args: read: Read function Returns: Tuple of (pack version, number of objects). If no data is available to read, returns (None, None). )NNNruPACKzInvalid pack header %rrw)ryrrx)rr )rheaderr num_objectsr!r!r"read_pack_headers  rcCs$t|trt|Sttt|SdSr%)r>rrsumr)chunksr!r!r" chunks_lengths rcCsL|dur |}|rd}nd}t||d\}}|dd?d@}|dd@}t|ddD] \} } || d@| dd>7}qVt|} |tkrt||d\}}| t|7} |d d @rt|dd@} |ddD] } | d7} | dK} | | d@7} q| } n2|tkr"|d } |rt| |}| d 7} nd} t|| ||}t ||||d }||fS) aUnpack a Git object. Args: read_all: Read function that blocks until the number of requested bytes are read. read_some: Read function that returns at least one byte, but may not return the number of bytes requested. compute_crc32: If True, compute the CRC32 of the compressed data. If False, the returned CRC32 will be None. include_comp: If True, include compressed data in the result. zlib_bufsize: An optional buffer size for zlib operations. Returns: A tuple of (unpacked, unused), where unused is the unused data leftover from decompression, and unpacked in an UnpackedObject with the following attrs set: * obj_chunks (for non-delta types) * pack_type_num * delta_base (for delta types) * comp_chunks (if include_comp is True) * decomp_chunks * decomp_len * crc32 (if compute_crc32 is True) Nr)rrurrrrr)rXrW) r# enumerater OFS_DELTAr REF_DELTArrr-r\)read_allrU compute_crc32rW zlib_bufsizerrtype_numrqrZbyteZraw_baseZdelta_base_offsetr3rVrZr!r!r" unpack_objectsB       rcCs$|\}}|tvrt|dSt|S)zBCompute the size of a unresolved object for use with LRUSizeCache.r)r7r)valuenumr&r!r!r"_compute_object_sizes rc@s\eZdZdZdefddZddZddZed d Z d d Z d dZ ddZ dddZ dS)PackStreamReaderz|Class to read a pack stream. The pack is read from a ReceivableProtocol using read() or recv() as appropriate. NcCsD||_|dur||_n||_t|_d|_t|_t|_||_ dSr) rrUrr:_offsetr_rbufr_trailer _zlib_bufsize)r(rrUrr!r!r"r)szPackStreamReader.__init__cs||}t|}j|7_tj}|dkr:|}d}nt||dd}|}jttfddt|Dj || dj|d| |S)aHRead up to size bytes using the given callback. As a side effect, update the verifier's hash (excluding the last 20 bytes read). Args: read: The read callback to read from. size: The maximum number of bytes to read; the particular behavior is callback-specific. rrcsg|]}jqSr!)rpopleft)rFrr9r!r"rH:rIz*PackStreamReader._read..N) rrrmaxr:r^r bytearrayrextend)r(rrqrMnZtnZto_popZto_addr!r9r"_read"s  zPackStreamReader._readcCs4|j}|}|dt|}||||Sr)rtellseekr )r(bufrrr!r!r"_buf_lenAs   zPackStreamReader._buf_lencCs|j|Sr%)rrr9r!r!r"r.IszPackStreamReader.offsetcCsD|}||kr|j|S|j}t|_|||j||S)z)Read, blocking until size bytes are read.)rrrrrr)r(rqbuf_lenZbuf_datar!r!r"rMs   zPackStreamReader.readcCs:|}|r,|j|}||kr(t|_|S||j|S)z7Read up to size bytes, blocking until one byte is read.)rrrrrrU)r(rqrrMr!r!r"recvVs zPackStreamReader.recvcCs|jSr% _num_objectsr9r!r!r"r`szPackStreamReader.__len__Fc cst|j\}|_|durdSt|jD]`}|j}t|j|j||jd\}}||_t}| || |j | d||_ |Vq&| dkr|dt |j}||jkrtt||jdS)aRead the objects in this pack file. Args: compute_crc32: If True, compute the CRC32 of the compressed data. If False, the returned CRC32 will be None. Returns: Iterator over UnpackedObjects with the following members set: offset obj_type_num obj_chunks (for non-delta types) delta_base (for delta types) decomp_chunks decomp_len crc32 (if compute_crc32 is True) Raises: ChecksumMismatch: if the checksum of the pack contents does not match the checksum in the pack trailer. zlib.error: if an error occurred during zlib decompression. IOError: if an error occurred writing to the output file. N)rUrrrr)rrrrr.rrrrwriterrrrrr:rr rr_) r(rZ pack_versionrr.rVrZrZpack_shar!r!r" read_objectscs*       zPackStreamReader.read_objects)F)r*r+r,rO _ZLIB_BUFSIZEr)rrrr.rrrrr!r!r!r"rs    rcs6eZdZdZd fdd ZfddZddZZS) PackStreamCopierzClass to verify a pack stream as it is being read. The pack is read from a ReceivableProtocol using read() or recv() as appropriate and written out to the given file-like object. Ncs$tt|j||d||_||_dS)aInitialize the copier. Args: read_all: Read function that blocks until the number of requested bytes are read. read_some: Read function that returns at least one byte, but may not return the number of bytes requested. outfile: File-like object to write output through. delta_iter: Optional DeltaChainIterator to record deltas as we read them. )rUN)rrr)outfile _delta_iter)r(rrUrZ delta_iterrr!r"r)s zPackStreamCopier.__init__cs"tt|||}|j||S)z:Read data from the read callback and write it to the file.)rrrrr)r(rrqrMrr!r"rs zPackStreamCopier._readcCs4|jr"|D]}|j|qn|D]}q*dS)zVerify a pack stream and write it to the output file. See PackStreamReader.iterobjects for a list of exceptions this may throw. N)rrrecord)r(rVrr!r!r"verifys   zPackStreamCopier.verify)N)r*r+r,rOr)rrrr!r!rr"rs rcCsLt}|t|t|t|tr0||n|D]}||q4|S)z5Compute the SHA for a numeric type and object chunks.)rr^rrr>rr)typerr:chunkr!r!r"r8s   r8cCst}|dt|}|dkr.|||ks6||krHtd|||f|||}|||r|t||}|||t|8}q^|S)aHash a portion of a file into a new SHA. Args: f: A file-like object to read from that supports seek(). start_ofs: The offset in the file to start reading at. end_ofs: The offset in the file to end reading at, relative to the end of the file. buffer_size: A buffer size for reading. Returns: A new SHA object updated with data read from the file. rzOAttempt to read beyond file length. start_ofs: %d, end_ofs: %d, file length: %d) rrr rrrminr^r)rgZ start_ofsend_ofsrXr:lengthtodorMr!r!r"compute_file_shas"     r c@seZdZdZd7ddZeddZeddZed8d d Z ed d Z d dZ ddZ ddZ ddZddZddZddZddZd9ddZd:d d!Zd"d#Zd;d$d%Zdd*d+Zd?d-d.Zd/d0Zd1d2Zd3d4Zd5d6ZdS)@PackDataaCThe data contained in a packfile. Pack files can be accessed both sequentially for exploding a pack, and directly with the help of an index to retrieve a specific object. The objects within are either complete or a delta against another. The header is variable length. If the MSB of each byte is set then it indicates that the subsequent byte is still part of the header. For the first byte the next MS bits are the type, which tells you the type of object, and whether it is a delta. The LS byte is the lowest bits of the size. For each subsequent byte the LS 7 bits are the next MS bits of the size, i.e. the last byte of the header contains the MS bits of the size. For the complete objects the data is stored as zlib deflated data. The size in the header is the uncompressed object size, so to uncompress you need to just keep feeding data to zlib until you get an object back, or it errors on bad data. This is done here by just giving the complete buffer from the start of the deflated object on. This is bad, but until I get mmap sorted out it will have to do. Currently there are no integrity checks done. Also no attempt is made to try and detect the delta case, or a request for an object at the wrong position. It will all just throw a zlib or KeyError. NcCsZ||_||_d|_|dur*t|jd|_n||_t|jj\}|_tdt d|_ d|_ dS)aaCreate a PackData object representing the pack in the given filename. The file must exist and stay readable until the object is disposed of. It must also stay the same size. It will be mapped whenever needed. Currently there is a restriction on the size of the pack as the python mmap implementation is flawed. rNrdi@)Z compute_size) rr _header_sizer rrrrrr _offset_cachepack)r(rrzrqrr!r!r"r)s zPackData.__init__cCstj|jSr%)rlrfbasenamerr9r!r!r"rszPackData.filenamecCs|jSr%rr9r!r!r"rfsz PackData.pathcCs|t|||dS)N)rzrq)str)clsrzrqr!r!r" from_file"szPackData.from_filecCs ||dS)N)rr!)rrfr!r!r" from_path&szPackData.from_pathcCs|jdSr%)rrr9r!r!r"r*szPackData.closecCs|Sr%r!r9r!r!r" __enter__-szPackData.__enter__cCs |dSr%rr(exc_typeZexc_valZexc_tbr!r!r"__exit__0szPackData.__exit__cCsft|tr||kSt|trbt|t|kr8dSt||D]\}}||krFdSqFdSdSr=)r>r rlistrr iterobjects)r(rAZo1Zo2r!r!r"rB3s  zPackData.__eq__cCsN|jdur|jStj|j|_|j|jkrHd|j|j|jf}t||jS)Nz(%s is too small for a packfile (%d < %d))rrlrfgetsizerrr)r(errmsgr!r!r" _get_size?s  zPackData._get_sizecCs|jS)z+Returns the number of objects in this pack.rr9r!r!r"rIszPackData.__len__cCst|jddS)z[Calculate the checksum for this pack. Returns: 20-byte binary SHA1 digest r)r )r rrr9r!r!r"rMszPackData.calculate_checksumcCs|jdurt|z|jj|}Wnty:d}Yn0|rP||\}}n,|jdurt|jjrt|j|\}}nt||||fS)z8Get the object for a ref SHA, only looking in this pack.N)rr~rr get_object_atresolve_ext_ref)r(r:r.rr&r!r!r"get_refTs   zPackData.get_refcCs|}|}|}g}|tvr|} |dur*|j}|tkr`|\} } || }||\}}t|tsJnF|tkr|\} } t| trt| dksJ|| \}}}t|tsJ| | || fq|} t |D]*\} }} t | | } | dur|| f|j | <q|| fS)z|Resolve an object, possibly resolving deltas when necessary. Returns: Tuple with object type and contents. Nr) r7r"rr r>intrrrrreversed apply_deltar)r(r.rr&r" base_offsetZ base_typeZbase_objZ delta_stackZ prev_offsetZ delta_offsetdeltarrZ delta_typer!r!r"resolve_objectes2 zPackData.resolve_objectTccs|j|jtd|jdD]`}|j}t|jj|d\}}|durT|||j||j| |j fV|jt | t qdS)Nrr) rrrrrrrrr2r<rrr)r(progressrrr.rVrZr!r!r"rs    zPackData.iterobjectsccsn|j|j|jdurdSt|jD]B}|j}t|jjdd\}}||_|V|jt | t q&dS)NFr)) rrrrrrrrr.rr)r(rr.rVrZr!r!r"_iter_unpackeds   zPackData._iter_unpackedccsX|j}|jdur|jjnd}tj||d}t|D] \}}|durL||||Vq2dS)zYield entries summarizing the contents of this pack. Args: progress: Progress function, called with current and total object count. Returns: iterator of tuples with (sha, offset, crc32) Nr!)rrr! PackIndexer for_pack_datar)r(r*rr!Zindexerrresultr!r!r"rs zPackData.iterentriescCst|j|d}|S)zReturn entries in this pack, sorted by SHA. Args: progress: Progress function, called with current and total object count Returns: List of tuples with (sha, offset, crc32) r*)sortedr)r(r*rr!r!r"sorted_entriesszPackData.sorted_entriescCsJ|j|d}t|d }t|||WdS1s<0YdS)zCreate a version 1 file for this data file. Args: filename: Index filename. progress: Progress report function Returns: Checksum of index file r0wbN)r2r write_pack_index_v1rr(rr*rrgr!r!r"create_index_v1s  zPackData.create_index_v1cCsJ|j|d}t|d }t|||WdS1s<0YdS)zCreate a version 2 index file for this data file. Args: filename: Index filename. progress: Progress report function Returns: Checksum of index file r0r3N)r2r write_pack_index_v2rr5r!r!r"create_index_v2s  zPackData.create_index_v2rycCs8|dkr|||S|dkr(|||Std|dS)zCreate an index file for this data file. Args: filename: Index filename. progress: Progress report function Returns: Checksum of index file rryzunknown index format %dN)r6r8rQ)r(rr*rr!r!r" create_indexs   zPackData.create_indexcCs|jdt|jdS)z1Return the expected checksum stored in this pack.rr)rrr rr9r!r!r"rszPackData.get_stored_checksumcCs&|}|}||kr"t||dS)z#Check the consistency of this pack.Nrrr!r!r"rszPackData.checkcCs>||jksJ|j|t|jjdd\}}|j|j|jfS)zGiven offset in the packfile return compressed data that is there. Using the associated index the location of an object can be looked up, and then the packfile can be asked directly for that object using this function. T)rW)rrrrrr2r3r4r(r.rVrr!r!r"get_compressed_data_ats  zPackData.get_compressed_data_atcCsXz |j|WStyYn0||jks.J|j|t|jj\}}|j|fS)zGiven an offset in to the packfile return the object that is there. Using the associated index the location of an object can be looked up, and then the packfile can be asked directly for that object using this function. ) rr~rrrrrr2r<r:r!r!r"r  s   zPackData.get_object_at)NN)N)N)NT)N)N)N)N)Nry)r*r+r,rOr)rrrf classmethodrrrrrrBrrrr"r(rr+rr2r6r8r9rrr;r r!r!r!r"r s@        *     r c@seZdZdZdZdZdddZedddZdd Z d d Z d d Z ddZ ddZ ddZddZddZddZddZdS)DeltaChainIteratoraAbstract iterator over pack data based on delta chains. Each object in the pack is guaranteed to be inflated exactly once, regardless of how many objects reference it as a delta base. As a result, memory usage is proportional to the length of the longest delta chain. Subclasses can override _result to define the result type of the iterator. By default, results are UnpackedObjects with the following members set: * offset * obj_type_num * obj_chunks * pack_type_num * delta_base (for delta types) * comp_chunks (if _include_comp is True) * decomp_chunks * decomp_len * crc32 (if _compute_crc32 is True) FNcCs6||_||_tt|_tt|_g|_i|_g|_dSr%) r_resolve_ext_refrr _pending_ofs _pending_ref _full_ofsZ_shas _ext_refs)r(Zfile_objr!r!r!r"r)4s  zDeltaChainIterator.__init__cCs2|d|d}|||D]}||q|S)Nr,) set_pack_datar+r)r pack_datar!ZwalkerrVr!r!r"r.=s     z DeltaChainIterator.for_pack_datacCs`|j}|j}|tkr0||j}|j||n,|tkrL|j|j|n|j||fdSr%) r2r.rr3r?rrr@rA)r(rVrr.r&r!r!r"rEs zDeltaChainIterator.recordcCs |j|_dSr%)r)r(rDr!r!r"rCPsz DeltaChainIterator.set_pack_dataccsL|jD]"\}}|||dD] }|Vqq|D] }|Vq2|jrHJdSr%)rA _follow_chain_walk_ref_chainsr?)r(r.rr/r!r!r"_walk_all_chainsSs   z#DeltaChainIterator._walk_all_chainscCs|jrtdd|jDdS)NcSsg|] }t|qSr!)rrEr!r!r"rH]rIz9DeltaChainIterator._ensure_no_pending..)r@r~r9r!r!r"_ensure_no_pending[sz%DeltaChainIterator._ensure_no_pendingc cs|js|dSt|jD]z\}}||jvr4q z||\}}Wnty\Yq Yn0|j||j||D]}| |||D] }|Vqqzq |dSr%) r>rHr1r@itemsr~rBrpoprE)r(Zbase_shapendingrr new_offsetr/r!r!r"rF_s       z#DeltaChainIterator._walk_ref_chainscCs|Sr%r!r(rVr!r!r"_resultvszDeltaChainIterator._resultcCsj|j|t|jj|j|jd\}}||_|durD|j|ksfJn"|jtvsRJ||_ t ||j |_ |S)N)rWr) rrrr _include_comp_compute_crc32r.r2r7r0r%r5r1)r(r.r0 base_chunksrVrr!r!r"_resolve_objectys   z"DeltaChainIterator._resolve_objectc#sv|||fg}|D]`\}}}|||||Vt|jjg|jg}|fdd|DqdS)Nc3s|]}|jjfVqdSr%)r0r1)rFrLrVr!r" sz3DeltaChainIterator._follow_chain..) rRrNrr?rJr.r@r:r)r(r.r0rQr Z unblockedr!rSr"rEs  z DeltaChainIterator._follow_chaincCs|Sr%)rGr9r!r!r"rszDeltaChainIterator.__iter__cCs|jSr%)rBr9r!r!r"ext_refsszDeltaChainIterator.ext_refs)N)N)r*r+r,rOrPrOr)r<r.rrCrGrHrFrNrRrErrUr!r!r!r"r=s    r=c@seZdZdZdZddZdS)r-z/Delta chain iterator that yields index entries.TcCs||j|jfSr%)r:r.rrMr!r!r"rNszPackIndexer._resultN)r*r+r,rOrPrNr!r!r!r"r-sr-c@seZdZdZddZdS) PackInflaterz1Delta chain iterator that yields ShaFile objects.cCs|Sr%)r;rMr!r!r"rNszPackInflater._resultN)r*r+r,rOrNr!r!r!r"rVsrVc@s:eZdZdZddZd ddZddZd d Zd d ZdS) SHA1ReaderAWrapper for file-like object that remembers the SHA1 of its data.cCs||_td|_dSNrI)rgrr(rgr!r!r"r)szSHA1Reader.__init__NcCs|j|}|j||Sr%)rgrrr^)r(rrMr!r!r"rs  zSHA1Reader.readcCs2|jd}||jkr.t|jt|dSr)rgrrrr r_r)r(rr!r!r" check_shas zSHA1Reader.check_shacCs |jSr%)rgrr9r!r!r"rszSHA1Reader.closecCs |jSr%rgrr9r!r!r"rszSHA1Reader.tell)N) r*r+r,rOr)rr[rrr!r!r!r"rWs  rWc@s@eZdZdZddZddZddZdd Zd d Zd d Z dS) SHA1WriterrXcCs||_d|_td|_dS)NrrI)rgr rrZr!r!r"r)szSHA1Writer.__init__cCs.|j||j||jt|7_dSr%)rr^rgrr rrLr!r!r"rs  zSHA1Writer.writecCs<|j}t|dksJ|j||jt|7_|Sr)rrrrgrr rr!r!r" write_shas   zSHA1Writer.write_shacCs|}|j|Sr%)r^rgrrr!r!r"rs zSHA1Writer.closecCs|jSr%)r r9r!r!r"r.szSHA1Writer.offsetcCs |jSr%r\r9r!r!r"rszSHA1Writer.tellN) r*r+r,rOr)rr^rr.rr!r!r!r"r]sr]cCsg}|d>|d@B}|dL}|r@||dB|d@}|dL}q|||tkr|d@g}|dL}|r|d8}|dd|d@B|dL}qd||n |tkrt|dksJ||7}t|S) a Create a pack object header for the given object info. Args: type_num: Numeric type of the object. delta_base: Delta base offset or ref, or None for whole objects. size: Uncompressed object size. Returns: A header for a packed object. rurrrrrrr)rrinsertrrrr)rr3rqrcrr!r!r"pack_object_headers(      rarc Csx|tvr|\}}nd}tt||t|}t||}d}||fD],} || |durb|| t | |}qB|d@S)a Write pack object to a file. Args: f: File to write to type: Numeric type of the object object: Object to write compression_level: the zlib compression level Returns: Tuple with offset at which the object was written, and crc32 NrrP) r7rrarrRcompressrr^rr) rgrobjectr:compression_levelr3rZ comp_datarrMr!r!r"write_pack_objects      recCst|dd&}t|||||d\}}Wdn1s:0Ytdd|D}t|dd }|t|||fWdS1s0YdS)aWrite a new pack data file. Args: filename: Path to the new pack file (without .pack extension) objects: Iterable of (object, path) tuples to write. Should provide __len__ window_size: Delta window size deltify: Whether to deltify pack objects compression_level: the zlib compression level Returns: Tuple with checksum of pack file and index file .packr3)delta_window_sizedeltifyrdNcSs"g|]\}}||d|dfqS)rrr!)rFkvr!r!r"rH.rIzwrite_pack...idx)r write_pack_objectsr1rIr7)robjectsrhrgrdrgrZdata_sumr!r!r" write_packs (rncCs2|d|tdd|td|dS)z4Write a pack header for the given number of objects.rrwryN)rr{r)rgrr!r!r"write_pack_header3s roccs|dur t}g}|D]"\}}||j|| |fq|t}|D]\}}}}|} | } d} |D]>} | j|krzqjt| | } t| t| krj| } | } qj|| | | fV| |t||krJ| qqJdS)aGenerate deltas for pack objects. Args: objects: An iterable of (object, path) tuples to deltify. window_size: Window size; None for default Returns: Iterator over type_num, object id, delta_base, content delta_base is None for full text entries N) DEFAULT_PACK_DELTA_WINDOW_SIZErrZ raw_lengthsortr as_raw_string create_deltarr:r appendleftrJ)rmZ window_sizemagicr&rfZpossible_basesrZ neg_lengthorawwinnerZ winner_basebaser'r!r!r"deltify_pack_objects:s,      rzcCst|}|dd|DfS)zCreate pack data from objects Args: objects: Pack objects Returns: Tuples with (type_num, hexdigest, delta base, object chunks) css,|]$\}}|j|d|fVqdSr%)rr:rrr)rFrvrfr!r!r"rTisz'pack_objects_to_data..)r)rmcountr!r!r"pack_objects_to_data`s r|cCs@|dur d}|r$t||}t|}n t|\}}t||||dS)aWrite a new pack data file. Args: f: File to write to objects: Iterable of (object, path) tuples to write. Should provide __len__ window_size: Sliding window size for searching for deltas; Set to None for default window size. deltify: Whether to deltify objects compression_level: the zlib compression level to use Returns: Dict mapping id -> (offset, crc32 checksum), pack checksum NFrd)rzrr|write_pack_data)rgrmrgrhrdZ pack_contentsZpack_contents_countr!r!r"rlms   rlc Csi}t|}t||t|D]\}\}}} } |durL|d||fd|} | durz|| \} } Wntyt}| | f} Yn0t}| | | f} t||| |d}| |f||<q|| fS)aWWrite a new pack data file. Args: f: File to write to num_records: Number of records records: Iterator over type_num, object_id, delta_base, raw progress: Function to report progress to compression_level: the zlib compression level Returns: Dict mapping id -> (offset, crc32 checksum), pack checksum Nzwriting pack data: %d/%d r]r}) r]rorr`r.r~rrrer^)rgZ num_recordsZrecordsr*rdrrrZ object_idr3rwr.r&Z base_crc32rr!r!r"r~s4    r~cCst|}tdd}|D]&\}}}|t|ddd7<qtdD]2}|td||||d||7<qH|D].\}}}|dkstd|td ||qt|d ksJ||| S) aWrite a new pack index file. Args: f: A file-like object to write to entries: List of tuples with object name (sha), offset_in_pack, and crc32_checksum. pack_checksum: Checksum of the pack file. Returns: The SHA of the written index file cSsdSrr!r!r!r!r"rIz%write_pack_index_v1..NrrrrPz)pack format 1 only supports offsets < 2Gbrr) r]rrrrr{r TypeErrorrr^)rgrr fan_out_tablerbr.entry_checksumrr!r!r"r4s    r4cCsHt}|d@}|dL}|r:||dB|d@}|dL}q|||S)Nrrr)rr)rqrr`r!r!r"_delta_encode_sizes  ricCsg}d}tdD]6}|d|d>@r|||d?d@|d|>O}qtdD]:}|d|d>@rP|||d?d@|dd|>O}qPt|g|S)Nrrurxrry)rrr)rr Zscratchoprr!r!r"_encode_copy_operations  rcCs2t|tsJt|tsJt}|tt|7}|tt|7}tj||d}|D]\}}}}}|dkr|} ||} | dkrt| t } |t | | 7}| | 7} | | 8} qz|dks|dkrX||} |} | dkr| d|t|| | d7}| d8} | d7} q| | |t|| | | 7}qXt|S)zUse python difflib to work out how to transform base_buf to target_buf. Args: base_buf: Base buffer target_buf: Target buffer )ar Zequalrreplacer_r) r>rrrrdifflibZSequenceMatcherZ get_opcodesr _MAX_COPY_LENrr)Zbase_bufZ target_bufZout_bufseqZopcodeZi1Zi2Zj1Zj2Z copy_startZcopy_lenZto_copyrGrvr!r!r"rss4      rsc Cst|tsd|}t|ts(d|}g}d}t|}dd}|||\}}|||\}}|t|ks|Jd|t|f||krt|||d}|d7}|d@rd} tdD]<} |d| >@rt|||d} |d7}| | | d >O} qd} td D]D} |dd| >@rt|||d} |d7}| | | d >O} q| dkrVd } | | | ks| | |ks| |krq||| | | q||dkr|||||||7}q|td q|||krtd ||d|t|krtd|S)zBased on the similar function in git's patch-delta.c. Args: src_buf: Source buffer delta: Delta instructions rIrcSsTd}d}|rLt|||d}|d7}||d@|>O}|d7}|d@sqLq||fS)Nrrirr)r)r'rrqrcmdr!r!r"get_delta_header_size%sz*apply_delta..get_delta_header_sizez%d vs %drrrurxrrzInvalid opcode 0zdelta not empty: %rNzdest size incorrect) r>rrKrrrrr r) Zsrc_bufr'outrZ delta_lengthrZsrc_sizeZ dest_sizerZcp_offrxZcp_sizer!r!r"r%sZ                 r%c Cslt|}|d|tddtdd}|D]&\}}}|t|ddd7<q4g}tdD]2}|td ||||d||7<qh|D]\}}}||q|D]\}}}|td |q|D]L\}}}|d kr|td |q|td d t|||q|D]}|td |q.t|d ksZJ||| S) aWrite a new pack index file. Args: f: File-like object to write to entries: List of tuples with object name (sha), offset_in_pack, and crc32_checksum. pack_checksum: Checksum of the pack file. Returns: The SHA of the index file written rvrrycSsdSrr!r!r!r!r"rgrIz%write_pack_index_v2..Nrrrwrs>Qr) r]rr{rrrrrrr^) rgrrrrbr.rZ largetablerr!r!r"r7Zs0        r7c@seZdZdZd1ddZeddZeddZd d Ze d d Z e d dZ ddZ ddZ ddZddZddZddZddZddZdd Zd!d"Zd#d$Zd%d&Zd'd(Zd)d*Zd+d,Zd-d.Zd2d/d0ZdS)3PackzA Git pack object.NcsP|_d_d_jd_jd_fdd_fdd_|_dS)Nrkrfcs tjSr%)r  _data_pathr!r9r!r"rrIzPack.__init__..cs tjSr%)rh _idx_pathr!r9r!r"rrI) _basename_data_idxrr _data_load _idx_loadr!)r(rr!r!r9r"r)s  z Pack.__init__cCs|d}||_||_|S)zTCreate a new pack object from callables to load pack data and index objects.)rr)rZdata_fnZidx_fnrr!r!r"from_lazy_objectsszPack.from_lazy_objectscCs4|d}||_||j_d|_||_d|_||S)z:Create a new pack object from pack data and index objects.rN)rrrrrcheck_length_and_checksum)rrMrrr!r!r" from_objectsszPack.from_objectscCs |jS)z2The SHA over the SHAs of the objects in this pack.)rrr9r!r!r"rbsz Pack.namecCs*|jdur$||_||j_||jS)z The pack data object being used.N)rrrrr9r!r!r"rMs   z Pack.datacCs|jdur||_|jS)zLThe index being used. Note: This may be an in-memory index N)rrr9r!r!r"rs  z Pack.indexcCs,|jdur|j|jdur(|jdSr%)rrrr9r!r!r"rs   z Pack.closecCs|Sr%r!r9r!r!r"rszPack.__enter__cCs |dSr%rrr!r!r"rsz Pack.__exit__cCst|t|o|j|jkSr%)r>rrrCr!r!r"rBsz Pack.__eq__cCs t|jS)zNumber of entries in this pack.)rrr9r!r!r"rsz Pack.__len__cCsd|jj|jfS)Nz%s(%r))rJr*rr9r!r!r"rNsz Pack.__repr__cCs t|jS)z7Iterate over all the sha1s of the objects in this pack.)rarr9r!r!r"rsz Pack.__iter__cCsJt|jt|jksJ|j}|j}||krFtt|t|dS)z@Sanity check the length and checksum of the pack index and data.N)rrrMrrr r)r(Zidx_stored_checksumZdata_stored_checksumr!r!r"rs  zPack.check_length_and_checksumcCs.|j|j|D] }|qdS)zCheck the integrity of this pack. Raises: ChecksumMismatch: if a checksum for the index or data is wrong N)rrrMrr'r!r!r"rs   z Pack.checkcCs |jSr%)rMrr9r!r!r"rszPack.get_stored_checksumcCs,z|j|WdSty&YdS0dS)z3Check whether this pack contains a particular SHA1.TFN)rrr~)r(rr!r!r" __contains__s   zPack.__contains__cCsH|j|}|j|\}}}|tkr>t|j||}t}|||fS)zGet raw unresolved data for a SHA. Args: sha1: SHA to return data for Returns: Tuple with pack object type, delta base (if applicable), list of data chunks )rrrMr;rrrr)r(rr.obj_typer3rr!r!r"get_raw_unresolveds  zPack.get_raw_unresolvedcCs>|j|}|j|\}}|j|||\}}|d|fSrY)rrrMr r(rK)r(rr.rr&rrr!r!r"get_raws z Pack.get_rawcCs||\}}tj|||dS)zRetrieve the specified SHA1.)r:)rrZfrom_raw_string)r(rrZuncompr!r!r" __getitem__ szPack.__getitem__cCsttj|j|jdS)z&Iterate over the objects in this pack.r,)rarVr.rMr!r9r!r!r"rszPack.iterobjectscCsGdddt}||S)zProvide an iterable for use with write_pack_objects. Returns: Object that can iterate over (object, path) tuples and provides __len__ c@s$eZdZddZddZddZdS)z+Pack.pack_tuples..PackTupleIterablecSs ||_dSr%)r)r(rr!r!r"r)sz4Pack.pack_tuples..PackTupleIterable.__init__cSs t|jSr%)rrr9r!r!r"r"sz3Pack.pack_tuples..PackTupleIterable.__len__cSsdd|jDS)Ncss|]}|dfVqdSr%r!)rFrvr!r!r"rT&rIzGPack.pack_tuples..PackTupleIterable.__iter__..)rrr9r!r!r"r%sz4Pack.pack_tuples..PackTupleIterable.__iter__N)r*r+r,r)rrr!r!r!r"PackTupleIterablesr)rc)r(rr!r!r" pack_tupless zPack.pack_tuplescCsPd|j}t|d(}|r.|||dWdn1sB0Y|S)a'Add a .keep file for the pack, preventing git from garbage collecting it. Args: msg: A message written inside the .keep file; can be used later to determine whether or not a .keep file is obsolete. Returns: The path of the .keep file, as a string. z%s.keepr3 N)rr r)r(msgZ keepfile_nameZkeepfiler!r!r"keep*s    (z Pack.keep)N)N)r*r+r,rOr)r<rrrbrrMrrrrrBrrNrrrrrrrrrrrr!r!r!r"rs8       r)r%r)N)N)rrr)Nr)NNr)N)NNr)Nr)YrO collectionsrriorrrrr{ itertoolsrrlsysZhashlibrrr r rRrp ImportErrorroplatformZdulwich.errorsr r Z dulwich.filer Zdulwich.lru_cacherZdulwich.objectsrrrrrrr7rpr# Exceptionr$rcr-rr\rcrhrtrerrrrrr}rrrrrrr8r r r=r-rVrWr]rarernrorzr|rlr~r4rrrrsr%r7Zwrite_pack_indexrZ dulwich._packr!r!r!r"s            O ?  Z% ) @ + 4     &   %,D&8