=OUjCXUddlZddlZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z ddlmZddlmZmZmZmZmZmZmZmZmZmZmZGddej8 ZGd d eZGd d eZdZ e e!dfe"d<ejFdddk(r e dz Z e dz Z dde de$fdZ%y)N)Module)IterableIteratorListOptionalTupleAnycast) LineNumbers) AstNodeToken TokenInfo match_tokenis_non_coding_tokenpatched_generate_tokens last_stmtannotate_fstring_nodesgenerate_tokens is_moduleis_stmtc eZdZdededdfdZej dedede e e e fe e e fffdZ d dedede e e ffd Z d dededefd Z y) ASTTextBase source_textfilenamereturnNcV||_t|}||_t||_yN) _filenamestr_textr _line_numbers)selfrrs d/workspace/moltex-experiments/.pixi/envs/default/lib/python3.12/site-packages/asttokens/asttokens.py__init__zASTTextBase.__init__s)DN k"KDJ$[1Dnodepaddedct) Returns two ``(lineno, col_offset)`` tuples for the start and end of the given node. If the positions can't be determined, or the nodes don't correspond to any particular text, returns ``(1, 0)`` for both. ``padded`` corresponds to the ``padded`` argument to ``ast.get_source_segment()``. This means that if ``padded`` is True, the start position will be adjusted to include leading whitespace if ``node`` is a multiline statement. )NotImplementedErrorr#r'r(s r$get_text_positionszASTTextBase.get_text_positions*s  r&c|j||\}}|jj||jj|fS)a Returns the (startpos, endpos) positions in source text corresponding to the given node. Returns (0, 0) for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r-r"line_to_offsetr#r'r(startends r$get_text_rangezASTTextBase.get_text_range9sP((v6JE3'd''/'d''- r&cJ|j||\}}|j||S)z Returns the text corresponding to the given node. Returns '' for nodes (like `Load`) that don't correspond to any particular text. See ``get_text_positions()`` for details on the ``padded`` argument. )r3r!r0s r$get_textzASTTextBase.get_textFs+$$T62JE3 ::eS !!r&)T)__name__ __module__ __qualname__r r%abcabstractmethodr boolrintr-r3r5r&r$rrs 2# 2 2 2  !%  U38_eCHo -.    $ %S/ "7"D"C"r&r) metaclassc eZdZdZ d(dededeededee e ddf fd Z d eddfd Z d e e de efd ZedefdZedeefdZedeefdZedefdZdedefdZdededefdZdededefdZd)dededefdZd)dededefdZ d*dededeededef d Z d)d!ed"edede efd#Zd)d$edede efd%Zd$ed&ede e eefe eefffd'Z!xZ"S)+ ASTTokensa ASTTokens maintains the text of Python code in several forms: as a string, as line numbers, and as tokens, and is used to mark and access token and position information. ``source_text`` must be a unicode or UTF8-encoded string. If you pass in UTF8 bytes, remember that all offsets you'll get are to the unicode text, which is available as the ``.text`` property. If ``parse`` is set, the ``source_text`` will be parsed with ``ast.parse()``, and the resulting tree marked with token info and made available as the ``.tree`` property. If ``tree`` is given, it will be marked and made available as the ``.tree`` property. In addition to the trees produced by the ``ast`` module, ASTTokens will also mark trees produced using ``astroid`` library . If only ``source_text`` is given, you may use ``.mark_tokens(tree)`` to mark the nodes of an AST tree created separately. Nrparsetreertokensrc|t||||rtj||n||_|t |j }t|j||_ |jDcgc]}|jc}|_ |jr|j|jyycc}wr) superr%astrA_treerr!list_translate_tokens_tokensstartpos_token_offsets mark_tokens)r#rrArBrrCtok __class__s r$r%zASTTokens.__init__es G[(+5:;1DJ~tzz*f..v67DL48<<@ddlm}||j|y)aq Given the root of the AST or Astroid tree produced from source_text, visits all nodes marking them with token and position information by adding ``.first_token`` and ``.last_token`` attributes. This is done automatically in the constructor when ``parse`` or ``tree`` arguments are set, but may be used manually with a separate AST or Astroid tree. r ) MarkTokensN)rMrR visit_tree)r#rPrRs r$rMzASTTokens.mark_tokens|s(t *r&original_tokensc#Ktt|D]a\}}|\}}}}}t|||||||jj |d|d|jj |d|dcyw)zS Translates the given standard library tokens into our own representation. rr N) enumeraterrr"r/) r#rTindexrNtok_typetok_strr1r2lines r$rIzASTTokens._translate_tokenss 7 HI s,/)hT (GUCu$$33E!HeAhG$$33CFCFC EEJsA:A<c|jS)z,The source code passed into the constructor.)r!r#s r$textzASTTokens.text ::r&c|jS)zIThe list of tokens corresponding to the source code from the constructor.)rJr\s r$rCzASTTokens.tokenss <<r&c|jS)zTThe root of the AST tree passed into the constructor or parsed from the source code.)rGr\s r$rBzASTTokens.treer^r&c|jS)zThe filename that was parsed)rr\s r$rzASTTokens.filenames >>r&offsetcb|jtj|j|dz S)z Returns the token containing the given character offset (0-based position in source text), or the preceeding token if the position is between tokens. r )rJbisectrL)r#rbs r$get_token_from_offsetzASTTokens.get_token_from_offsets) << d&9&96BQF GGr&lineno col_offsetcX|j|jj||S)z Returns the token containing the given (lineno, col_offset) position, or the preceeding token if the position is between tokens. )rer"r/r#rfrgs r$ get_tokenzASTTokens.get_tokens)  % %d&8&8&G&GPZ&[ \\r&cZ|j||jj||S)zd Same as get_token(), but interprets col_offset as a UTF8 offset, which is what `ast` uses. )rjr" from_utf8_colris r$get_token_from_utf8zASTTokens.get_token_from_utf8s( >>&$"4"4"B"B6:"V WWr&rN include_extrac|jdz}|sJt|j|jr(|dz }t|j|jr(|j|S)z Returns the next token after the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rWrrJtyper#rNrnis r$ next_tokenzASTTokens.next_tokenY A A   Q 4 4 5 Q  Q 4 4 5 <<?r&c|jdz }|sJt|j|jr(|dz}t|j|jr(|j|S)z Returns the previous token before the given one. If include_extra is True, includes non-coding tokens from the tokenize module, such as NL and COMMENT. r rprrs r$ prev_tokenzASTTokens.prev_tokenrur& start_tokenrXrYreversec|}|r |jn |j}t|||sVtj|j s7||d}t|||s tj|j s7|S)z Looks for the first token, starting at start_token, that matches tok_type and, if given, the token string. Searches backwards if reverse is True. Returns ENDMARKER token if not found (you can check it with `token.ISEOF(t.type)`). Trn)rwrtrtokenISEOFrq)r#rxrXrYrytadvances r$ find_tokenzASTTokens.find_tokensc A!(doodooG!Xw/ AFF8K !4 (a!Xw/ AFF8K Hr& first_token last_tokenc#Kt|j|jdzD]8}|s#t|j|jr(|j|:yw)z Yields all tokens in order from first_token through and including last_token. If include_extra is True, includes non-coding tokens such as tokenize.NL and .COMMENT. r N)rangerWrrJrq)r#rrrnrss r$ token_rangezASTTokens.token_rangesQ;$$j&6&6&: ; 1$,,q/2F2FGll1oz/ASTTokens.get_text_positions..sS=Rk!U]]3=Rs$&r)hasattrrr1rr2anyrr0s r$r-zASTTokens.get_text_positionssa 4 '     " "E //  C #ST__T=RSSQxme #:r&)FN N)F)NF)#r6r7r8__doc__r r;rrr rrr%rMrrrIpropertyr]rrCrBrr<rerjrmrtrwrrr rrr- __classcell__rOs@r$r@r@QsZ,!,0 ## # 6  #  # Xi( ) # #. +6 +d +Ex /BExPUE C  d5k  HV$   H#H%H]c]s]u]XXXX E $ 5  E $ 5 ]b    (+  6>sm  UY     $      \W\T\huo\WduU3PS8_V[\_ad\dVeEe?fr&r@c eZdZdZddedeededdffd Ze defdZ e de fd Z d e d edeeeefeeefffd Zd e d edeeeefeeefffd ZxZS)ASTTextaF Supports the same ``get_text*`` methods as ``ASTTokens``, but uses the AST to determine the text positions instead of tokens. This is faster than ``ASTTokens`` as it requires less setup work. It also (sometimes) supports nodes inside f-strings, which ``ASTTokens`` doesn't. Some node types and/or Python versions are not supported. In these cases the ``get_text*`` methods will fall back to using ``ASTTokens`` which incurs the usual setup cost the first time. If you want to avoid this, check ``supports_tokenless(node)`` before calling ``get_text*`` methods. NrrBrrct|||||_|jt|jd|_yr)rEr%rGr _asttokens)r#rrBrrOs r$r%zASTText.__init__s6 G[(+DJ zzTZZ(+/DOr&c|jDtj|j|j|_t |j|jSr)rGrFrAr!rrr\s r$rBz ASTText.tree!s< zz99TZZ8djTZZ( ::r&c|j1t|j|j|j|_|jS)N)rBr)rr@r!rBrr\s r$ asttokenszASTText.asttokens(s9 ! **yy>>do ??r&r'r(ct|r0d|jjt|jfSt |ddy|sJt |dg}|st |dd}t |dg}|r|d}n|}|j }t|}|r9||j k7s'||jk7rt |d drt|rd}n&|jj||j}||f} tt|j} tt|j} |jj| | } | | f} | | fS) zF Version of ``get_text_positions()`` that doesn't use tokens. rrfNrdecorator_list decoratorsnodesrdoc_node)rr"offset_to_linelenr!getattrrfr end_linenorrlrgr r<end_col_offset) r#r'r(rdecorators_node start_node start_linenoend_nodestart_col_offsetr1rrr2s r$_get_text_positions_tokenlessz%ASTText._get_text_positions_tokenless2sU  T''66s4::G GGtXt$,  K4/4J lD9o?GR8ja=jj$$LH' DOO +j$/ ++99, H]H]^ + ,E c8../J#x667N''55j.QN ~ &C #:r&ct|ddryt|r|j||S|jj ||S)r*_broken_positionsNr)rsupports_tokenlessrrr-r,s r$r-zASTText.get_text_positionsusFt($/ $  / /f == >> , ,T6 ::r&)Nr)r6r7r8rr rrr r%rrBr@rr r;rr<rr-rrs@r$rr s 0#0Xf-=0PS0fj0 F    AA!%A U38_eCHo -.AF;W;d;uU3PS8_V[\_ad\dVeEe?f;r&r) arguments Argumentswithitem._unsupported_tokenless_types))argStarred)SliceExtSliceIndexkeywordr'rc>t|jtvxrt|tj  xrA|duxr;t|jdk(xr!t|j jdv xr dtjjvS)av Returns True if the Python version and the node (if given) are supported by the ``get_text*`` methods of ``ASTText`` without falling back to ``ASTTokens``. See ``ASTText`` for why this matters. The following cases are not supported: - PyPy - ``ast.arguments`` / ``astroid.Arguments`` - ``ast.withitem`` - ``astroid.Comprehension`` - ``astroid.AssignName`` inside ``astroid.Arguments`` or ``astroid.ExceptHandler`` - The following nodes in Python 3.8 only: - ``ast.arg`` - ``ast.Starred`` - ``ast.Slice`` - ``ast.ExtSlice`` - ``ast.Index`` - ``ast.keyword`` N AssignName)r ExceptHandlerpypy) rqr6r isinstancerFASTparentsysversionlower)r's r$rrs, 4j!==, tSWW %% $d*: J  < / MT[[!**.LL  ,  ))+ + r&r)&r9rFrdrr|rtypingrrrrrr r line_numbersr utilr rrrrrrrrrrABCMetarr@rrr __annotations__ version_infor;rr=r&r$rs GGG% 0"CKK0"fw wt{;k{;@1WeCHoVBQ6!"44"KKSDr&