:vj.dZddlmZddlZddlZddlZddlZddlmZm Z ddl m Z ddl m Z ddlmZmZmZddlZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3ddl4m5Z5m6Z6erddl7m8Z8Gdde-Z9GddeZ:Gdde1Z;Gdde.Z<ejze<Gd d!e Z>Gd"d#e>Z?Gd$d%eZ@ejze@y)&zKernel gateway managers.) annotationsN)EmptyQueue)Thread) monotonic) TYPE_CHECKINGAnycast)AsyncKernelClient)KernelClientABC)KernelSpecManager)KernelManagerABC) ensure_async)web) json_decode json_encode url_escapeutf8)DottedObjectNameInstanceTypedefault)UTCutcnow)AsyncMappingKernelManagerServerKernelManageremit_kernel_action_event)SessionManager) url_path_join) GatewayClientgateway_request)LoggerceZdZUdZiZded<eddZeddZfdZ d Z d d d d Z d Z dZ ddZddZdZddZfdZxZS)GatewayMappingKernelManagerz[Kernel manager that supports remote kernels hosted by Jupyter Kernel or Enterprise Gateway.zdict[str, GatewayKernelManager]_kernelskernel_manager_classcy)Nz4jupyter_server.gateway.managers.GatewayKernelManagerselfs p/workspace/moltex-experiments/.pixi/envs/default/lib/python3.12/site-packages/jupyter_server/gateway/managers.py_default_kernel_manager_classz9GatewayMappingKernelManager._default_kernel_manager_class.sEshared_contextcyNFr*r+s r-_default_shared_contextz3GatewayMappingKernelManager._default_shared_context2r/c t|di|ttjj xsdtjj xsd|_y)z,Initialize a gateway mapping kernel manager.Nr*)super__init__r r"instanceurlkernels_endpoint kernels_urlr,kwargs __class__s r-r8z$GatewayMappingKernelManager.__init__6sN "6"(  " " $ ( ( .B 0F0F0H0Y0Y0_]_ r/cX |jj|S#t$rYywxYw)zCComplete override since we want to be more tolerant of missing keysN)r'popKeyError)r, kernel_ids r- remove_kernelz)GatewayMappingKernelManager.remove_kernel=s. ==$$Y/ /   s  ))N)rCpathcfK|jjd|d|d|||j||d<|j||j}|jdd|i|d{|j }||j |<|js|j|S7=w) aStart a kernel for a session and return its kernel_id. Parameters ---------- kernel_id : uuid The uuid to associate the new kernel with. If this is not None, this kernel will be persistent whenever it is requested. path : API path The API path (unicode, '/' delimited) for the cwd. Will be transformed to an OS path relative to root_dir. z Request start kernel: kernel_id=z, path=''Ncwd)parentlogrCr*) rJinfo cwd_for_pathkernel_manager_factory start_kernelrCr'_initialized_cullerinitialize_culler)r,rCrEr>kms r-rNz(GatewayMappingKernelManager.start_kernelDs  8 8D6QRST  !1 --d3F5M  ( ($(( ( Cboo< B1cbKd}|jt|}|r |j}|Sw)zReturn a dictionary of kernel information described in the JSON standard model. Parameters ---------- kernel_id : uuid The uuid of the kernel. N) get_kernelstrkernel)r,rCmodelrQs r- kernel_modelz(GatewayMappingKernelManager.kernel_model`s0 __S^ , IIE s-/cJK|jjd|jt|jdd{}t |j }i}|D]A}|d}||j vs|j |j|d{|||<C|j j}g}|D]}||vs|jjd|d |j |jd{}|r|||<U|jjd|d|j j|d|j|t|jS7<77x#tj$rd}YwxYww) zGet a list of running kernels from the Gateway server. We'll use this opportunity to refresh the models in each of the kernels we're managing. zRequest list kernels: GETmethodNidzKernel zH not present in the list of kernels - possibly culled on Gateway server.z6 no longer active - probably culled on Gateway server.)rJdebugr<r#rbodyr' refresh_modelcopywarningr HTTPErrorrAappendlistvalues) r,r>responsekernels kernel_modelsrVkid our_kernels culled_idss r- list_kernelsz(GatewayMappingKernelManager.list_kernelsos /0@0@/ABC()9)9%HHhmm, E+Cdmm#mmC(66u===%* c"  mm((*  C-'  cU"jk!#'--"4"B"B"DDE).M#&HH$$!#&\]MM%%c40%%c*;<M((*++WI>0E}}! E!sgAF#F3F#:!F#F0F# F#- F FFA0F#F#FF F#F  F#cK|j|}t|j||d{|j|y7w)aXShutdown a kernel by its kernel uuid. Parameters ========== kernel_id : uuid The id of the kernel to shutdown. now : bool Shutdown the kernel immediately (True) or gracefully (False) restart : bool The purpose of this shutdown is to restart the kernel (True) )nowrestartN)rSrshutdown_kernelrD)r,rCrnrorQs r-rpz+GatewayMappingKernelManager.shutdown_kernelsH__Y '2--#w-GHHH 9% Is1A A A c|K|j|}t|jdd|i|d{y7w)zRestart a kernel by its kernel uuid. Parameters ========== kernel_id : uuid The id of the kernel to restart. rnNr*)rSrrestart_kernel)r,rCrnr>rQs r-rrz*GatewayMappingKernelManager.restart_kernels:__Y ',2,,???@@@s 2<:<ctK|j|}t|jd{y7w)zInterrupt a kernel by its kernel uuid. Parameters ========== kernel_id : uuid The id of the kernel to interrupt. N)rSrinterrupt_kernel)r,rCr>rQs r-rtz,GatewayMappingKernelManager.interrupt_kernels-__Y '2..0111s .868cKt|j}|D]G}|j|}t|j |d{|j |Iy7w)zShutdown all kernels.)rnN)rdr'rSrrprD)r,rnkidsrCrQs r- shutdown_allz(GatewayMappingKernelManager.shutdown_allsYDMM"I+Br11c1:; ; ;   y ) ;sA A' A% A'crK|jd{t| d{y77w)z@Override cull_kernels, so we can be sure their state is current.N)rlr7 cull_kernelsr,r?s r-ryz(GatewayMappingKernelManager.cull_kernelss3!!!g"$$$ "$s737577FFF)__name__ __module__ __qualname____doc__r'__annotations__rr.r3r8rDrNrWrlrprrrtrwry __classcell__r?s@r-r&r&(se13H-2 #$F%F  /38 2,h& A 2*%%r/r&cXeZdZdZfdZedZdZd dZdZ dZ dZ d Z xZ S) GatewayKernelSpecManagerzA gateway kernel spec manager.c t|di|ttjj xsdtjj }tj||_ ttjj xsdtjj|_ y)z)Initialize a gateway kernel spec manager.r6Nr*) r7r8r r"r9r:kernelspecs_endpointr_get_endpoint_for_user_filter base_endpointkernelspecs_resource_endpointbase_resource_endpoint)r,r>rr?s r-r8z!GatewayKernelSpecManager.__init__s "6"%  " " $ ( ( .B 0F0F0H0]0] 6SSTab&3  " " $ ( ( .B  " " $ B B' #r/cVtjjd}|r|d|S|S)z#Get the endpoint for a user filter.KERNEL_USERNAMEz?user=)osenvironget)default_endpoint kernel_users r-rz6GatewayKernelSpecManager._get_endpoint_for_user_filters2jjnn%67 &'vk]; ;r/c z|jsiS|d}|D]}||d}|D]}||}tj|dd}t|dkDs/t |jj d|d}||d|d|<||k7sg|j jd|d|d|d||S)zHelper method that replaces any gateway base_url with the server's base_url This enables clients to properly route through jupyter_server to a gateway for kernel resources such as logo files kernelspecs resourcesz /kernelspecs/r!)sepmaxsplitz'Replaced original kernel resource path z with new path )rIrTrsplitlenr base_urlrJr]) r, kernel_specsr kernel_namer resource_name original_pathsplit_eg_base_urlnew_paths r-"_replace_path_kernelspec_resourcesz;GatewayKernelSpecManager._replace_path_kernelspec_resourcess {{I"=1 &K#K0=I!* )- 8 $'JJ}/\]J$^!()A-, ,,m=Nq=Q H\dL/ <[I-X$0Em_U$$0$? $L[$YZg$h#ik"+'r/c\|rt|jt|S|jS)zBuilds a url for the kernels endpoint Parameters ---------- kernel_name : kernel name (optional) )r rr)r,rs r-_get_kernelspecs_endpoint_urlz6GatewayKernelSpecManager._get_kernelspecs_endpoint_urls+  !3!3Z 5LM M!!!r/cRK|jd{}|jsiS|jj}|jd}||jk7r3|j j d|d|jd||_|jd}|S7w)z,Get all of the kernel specs for the gateway.Nrz'Default kernel name on Gateway server (z ) differs from Notebook server (z'). Updating to Gateway server's value.r)list_kernel_specsrIkernel_managerrdefault_kernel_namerJrK)r,fetched_kspecsrQremote_default_kernel_name remote_kspecss r- get_all_specsz&GatewayKernelSpecManager.get_all_specss#5577 {{I [[ ' '%3%7%7 %B" %)?)? ? HHMM9:T9UV$$&$:$:#;;bd &@B "&**=9 %8sB'B%BB'cK|j}|jjd|t|dd{}t |j }|j |}|S7,w)zGet a list of kernel specs.zRequest list kernel specs at: rYrZN)rrJr]r#rr^r)r,kernel_spec_urlrfrs r-rz*GatewayKernelSpecManager.list_kernel_specs)sf<<> 77HIJ(GG"8==1 >>|L Hs?A0A.-A0cK|jt|}|jjd| t |dd{}t |j }|S7#tj$rE}|jdk(r0d|dtjj}t|dd}~wwxYww) zGet kernel spec for kernel_name. Parameters ---------- kernel_name : str The name of the kernel. )rzRequest kernel spec at: rYrZNz kernelspec z! not found on Gateway server at: )rrTrJr]r#rr^rrb status_coder"r9r:rB)r,rr>rrf kernel_specerrormsgs r-get_kernel_specz(GatewayKernelSpecManager.get_kernel_spec2s<<[IY<Z 1/1BCD 5,_UKKH&hmm4KL}}   C'$K=0QR_RhRhRjRnRnQopsm- s;:CA+ A)A+C)A++C>AB>>CCcRKt|jt|t|}|jj d|d| t |dd{}|j }|S7#tj$r}|jdk(rd}nYd}~|Sd}~wwxYww)zGet kernel spec for kernel_name. Parameters ---------- kernel_name : str The name of the kernel. path : str The name of the desired resource zRequest kernel spec resource 'z' at: rYrZNr) r rrTrJr]r#r^rrbr)r,rrEkernel_spec_resource_urlrfkernel_spec_resourcers r-get_kernel_spec_resourcez1GatewayKernelSpecManager.get_kernel_spec_resourceKs$1  ' '[)93t9$   7vVD\C]^_ 1,-EeTTH$,== ##U}}   C''+$% $# sBA B'A3A1A3#B'1A33B$BB'B$$B'N)r}r~rrr8 staticmethodrrrrrrrrrs@r-rrs=(    2 ",2$r/rc(eZdZdZedZddZy)GatewaySessionManagerzA gateway session manager.z;jupyter_server.gateway.managers.GatewayMappingKernelManagercrKd} |jj|}|duS#t$rY|duSwxYww)zRChecks if the kernel is still considered alive and returns true if it's not found.N)rrS Exception)r,rCrQs r- kernel_culledz#GatewaySessionManager.kernel_culledjsQ*. $$// :BTz  Tz s7%7 4747N)rCrTreturnbool)r}r~rrrrrr*r/r-rres$[\Nr/rceZdZUdZdZded<dZeddZfdZ e dZ e d Z ed Zd Zdd Zed dZedddZeddZeddZdZddZxZS)GatewayKernelManagerz6Manages a single kernel remotely via a Gateway Server.N str | NonerC cache_portscyr2r*r+s r-_default_cache_portsz)GatewayKernelManager._default_cache_portsr4r/c  t|di|ttjj xsdtjj |_|dx|_|_ d|_ t|_ y)z&Initialize the gateway kernel manager.r6Nstartingr*) r7r8r r"r9r:r;r<rUrCexecution_stater last_activityr=s r-r8zGatewayKernelManager.__init__sm "6"(  " " $ ( ( .B 0F0F0H0Y0Y  '++ dn)#Xr/c|jduS)z/Has a kernel been started that we are managing.N)rUr+s r- has_kernelzGatewayKernelManager.has_kernels{{$&&r/z3jupyter_server.gateway.managers.GatewayKernelClient)klassc i}|j|jd|j|j|d|j|d<|j||jdi|S)z3Create a client configured to connect to our kernelT)session)connection_filerIrCr*)updateget_connection_inforrCclient_factory)r,r>kws r-clientzGatewayKernelManager.clientsq $**4*89 #'#7#7  ..; &"t""(R((r/cK|{|jjd|jz t|jdd{}t |j }|jjd|z|rtjj|dd jt |_|d |_t!|j"t$r/t'|d |j"j(|j*<||_|S7#t j$rF}|jdk(r+|jjd|jzd}nYd}~d}~wwxYww) zRefresh the kernel model. Parameters ---------- model : dict The model from which to refresh the kernel. If None, the kernel model is fetched from the Gateway server. NzRequest kernel at: %srYrZrzKernel not found at: %szKernel retrieved: %srz%Y-%m-%dT%H:%M:%S.%fZ)tzinfor connections)rJr] kernel_urlr#rr^rrbrradatetimestrptimereplacerrr isinstancerIrint_kernel_connectionsrCrU)r,rVrfrs r-r_z"GatewayKernelManager.refresh_models< = HHNN2T__D E 3!0!OO$HMM2 HHNN1E9 : !)!2!2!;!;o&(?"gSg!  $)):#;D $++'@A CFeMFZB[ //?  3P== $$+HH$$%>%PQ E sA+E8DD D C E8DE5/;E0*E80E55E8zKernel {kernel_id} was started.) success_msgcK|jd}|8|jdd}|jjd|jztj jdQt jjr3t jjxsdtj d<tj j}|j|jdi|jDcic]H\}}|jd s/|t jjjd vr||J}}}|jd |jd |d |d <t||d }t!|jdddi|d{} t#| j$|_|j&d|_t+|jt-t/|j(|_|jj3d|j(d|y||_t+|jt-t/|j(|_|j5d{|_|jj3d|j(ycc}}w7 7;w)aStarts a kernel via HTTP in an asynchronous manner. Parameters ---------- `**kwargs` : optional keyword arguments that are passed down to build the kernel_cmd and launching the kernel (e.g. Popen kwargs). rCNrpython3zRequest new kernel at: %srr6envKERNEL_,rHKERNEL_WORKING_DIR)namerPOST Content-Typeapplication/jsonr[headersr^r\z%GatewayKernelManager started kernel: z, args: z,GatewayKernelManager using existing kernel: )rrJr]r<rrr"r9 http_userr`ritems startswith allowed_envssplitrr#rr^rUrCr rrTrrKr_) r,r>rCr payload_envskv kernel_env json_bodyrfs r-rNz!GatewayKernelManager.start_kernelsUJJ{+   **]I>K HHNN69I9II Jzz~~/08]=S=S=U=_=_0=0F0F0H0R0R0XVX ,-::??,L    5" 5 6 +0022FQ<< *a=3I3I3K3X3X3^3^_b3c.c12 zz% ,@T1U1]39%= /0#[$LMI,  ');< H &hmm4DK![[.DN+D,<,]^DO HHMMA$..AQQYZ`Yab c&DN+D,<,]^DO $ 2 2 44DK HHMMHHXY Z55s3DKA K AK9K:C&K K!9KKz Kernel {kernel_id} was shutdown.cK|jrx|jjd|j t |jdd{}|jjd|j |j yy77#tj$r6}|jdk(r|jjdnYd}~yd}~wwxYww)z5Attempts to stop the kernel process cleanly via HTTP.zRequest shutdown kernel at: %sDELETErZNzShutdown kernel response: %d %srz4Shutdown kernel response: kernel not found (ignored)) rrJr]rr#codereasonrrbr)r,rnrorfrs r-rpz$GatewayKernelManager.shutdown_kernels ?? HHNN;T__ M !0!RR@(--QYQ`Q`a S== $$+HHNN#YZ[ s@3CB B5B CB C,C CCCz!Kernel {kernel_id} was restarted.cBK|jr|jJ|jdz}|jjd|t |dddit id{}|jjd|j |jyy77w) zRestarts a kernel via HTTP.Nz/restartzRequest restart kernel at: %srrrrzRestart kernel response: %d %srrrJr]r#rrr)r,rrrfs r-rrz#GatewayKernelManager.restart_kernel s ????. ..:5J HHNN:J G,');< _ H HHNN;X]]HOO \ A#B%B&8Bz#Kernel {kernel_id} was interrupted.cBK|jr|jJ|jdz}|jjd|t |dddit id{}|jjd|j |jyy77w) z*Interrupts the kernel via an HTTP request.Nz /interruptzRequest interrupt kernel at: %srrrrz Interrupt kernel response: %d %sr)r,rrfs r-rtz%GatewayKernelManager.interrupt_kernel1s ????. ..<7J HHNNr/rc~eZdZUdZdZded<ded<dfd ZddZdd Zdd Z e d Z dd Z dd Z ddZxZS) ChannelQueuezA queue for a named channel.Nr channel_namerresponse_router_finishedcZt|||_||_||_d|_y)zInitialize a channel queue.FN)r7r8r channel_socketrJr )r,r r rJr?s r-r8zChannelQueue.__init__Zs- (,(-%r/c8K| td}n|dkr d}t|t|z} |jdS#t$rI|j rd}t |dt|kDrtjdd{7YnwxYwiw)z"Asynchronously get from the queue.Ninfrz''timeout' must be a non-negative numberF)blockzResponse router had finished) float ValueErrorrrrr  RuntimeErrorasynciosleep)r,timeoutrend_times r- _async_getzChannelQueue._async_getbs ?ElG q[;CS/ !;( 'xxex,, '008C&s+5;)mmA&&&  's5/BABA B BBBBBcK|jdd}|j|d{}|jjd|j|d|r|dnd|j t d |S7Rw) zGet a message from the queue.rr!)rNz9Received message on channel: %s, msg_id: %s, msg_type: %smsg_idmsg_typenulldict[str, Any])rrrJr]r  task_doner )r,argsr>rrs r-get_msgzChannelQueue.get_msgvsv**Y*OOGO44  G    M"C O  $c**5s(A?A=AA?ctj|tjj dd}|j j d|j|d|r|dnd|jj|y) zSend a message to the queue.)rz !r/cy)zStart the queue.Nr*r+s r-startzChannelQueue.startrr/c b|jsg}|jr=|j}|ddk7r|j|d|jr=|jdk(rd|vry|r9|j j d|jdt|d|d yyy) zStop the queue.rstatusiopubshutdown_replyNzStopping channel 'z' with z" unprocessed non-status messages: .)emptyqsize get_nowaitrcr rJrar)r,msgsrs r-stopzChannelQueue.stopszz|D**,oo'z?h.KKJ0**,  G+0@D0H  ():):(;73t9+Moptouuvwr/c|jduS)zWhether the queue is alive.N)r r+s r-rzChannelQueue.is_alives""$..r/)r rTr zwebsocket.WebSocketrJr$r)rr r>r rr)rrrNone)rr6rr)r}r~rrr rr8rrr$rr#r*r4rrrs@r-rrTsM&#L*#"".'( + * $/r/rceZdZdZddZy)HBChannelQueuez"A queue for the heartbeat channel.c"|jS)zWhether the channel is beating.)rr+s r- is_beatingzHBChannelQueue.is_beatings}}r/Nr7)r}r~rrr;r*r/r-r9r9s ,r/r9ceZdZUdZdZded<ded<ded<ded <ded <ded <ded <fd Zdfd ZfdZe dZ e dZ e dZ e dZ e dZdZxZS)GatewayKernelClientaCommunicates with a single kernel indirectly via a websocket to a gateway server. There are five channels associated with each kernel: * shell: for request/reply calls to the kernel. * iopub: for the kernel to publish results to frontends. * hb: for monitoring the kernel's heartbeat. * stdin: for frontends to reply to raw_input calls in the kernel. * control: for kernel management calls to the kernel. The messages that can be sent on these channels are exposed as methods of the client (KernelClient.execute, complete, history, etc.). These methods only send the message, they don't wait for a reply. To get results, use e.g. :meth:`get_shell_msg` to fetch messages from the shell channel. Fr_channels_stoppedzdict[str, ChannelQueue] | None_channel_queueszChannelQueue | None_control_channel _hb_channel_stdin_channel_iopub_channel_shell_channelc jt|di|||_d|_d|_d|_i|_y)z#Initialize a gateway kernel client.NFr*)r7r8rCr response_routerr>r?)r,rCr>r?s r-r8zGatewayKernelClient.__init__s: "6"":>.2!&!r/c Kttjjxsdtjjt |j d}tjjtjjtjjd}tj|tjjd||_ tt|A|||||d{t#|j$|_|j&j)y7:w) zStarts the channels for this kernel. For this class, we establish a websocket connection to the destination and set up the channel-based queues on which applicable messages will be posted. r6channels)ca_certscertfilekeyfileT)renable_multithreadsslopt)shellr-stdinhbcontrolN)target)r r"r9ws_urlr;rrCrI client_cert client_key websocketcreate_connectionKERNEL_LAUNCH_TIMEOUTr rr7start_channelsr_route_responsesrFr*) r,rNr-rOrPrQrS ssl_optionsr?s r-rYz"GatewayKernelClient.start_channelss  " " $ + + 1r  " " $ 5 5 t~~ &   &..099%..0<<$--/:: (99 !**,BB#   G "e5RY` " a    &T-B-BC ""$  sDEE;Ecbt|d|_|jj d|j J|j j |jJ|jj|jr"|jjd|_ yy)zStops all the running channels for this kernel. For this class, we close the websocket connection and destroy the channel-based queues. TzClosing websocket connectionN) r7 stop_channelsr>rJr]r closerFjoinr?clearrzs r-r]z!GatewayKernelClient.stop_channelss !% 56""... !!###/// !!#     & & (#'D  r/c|jv|jjd|jJt d|j|j|_|j J|j|j d<|jS)z-Get the shell channel object for this kernel.zcreating shell channel queuerN)rDrJr]r rr?r+s r- shell_channelz!GatewayKernelClient.shell_channel    & HHNN9 :&&2 22".w8K8KTXX"VD ''3 33,0,?,?D  )"""r/c|jv|jjd|jJt d|j|j|_|j J|j|j d<|jS)z-Get the iopub channel object for this kernel.zcreating iopub channel queuer-)rCrJr]r rr?r+s r- iopub_channelz!GatewayKernelClient.iopub_channel#rcr/c|jv|jjd|jJt d|j|j|_|j J|j|j d<|jS)z-Get the stdin channel object for this kernel.zcreating stdin channel queuerO)rBrJr]r rr?r+s r- stdin_channelz!GatewayKernelClient.stdin_channel.rcr/c|jv|jjd|jJt d|j|j|_|j J|j|j d<|jS)z*Get the hb channel object for this kernel.zcreating hb channel queuerP)rArJr]r r9r?r+s r- hb_channelzGatewayKernelClient.hb_channel9s    # HHNN6 7&&2 22-dD4G4GRD ''3 33)-)9)9D  &r/c|jv|jjd|jJt d|j|j|_|j J|j|j d<|jS)z/Get the control channel object for this kernel.zcreating control channel queuerQ)r@rJr]r rr?r+s r-control_channelz#GatewayKernelClient.control_channelDs  ( HHNN; <&&2 22$0Dr recvrrr? put_nowaitrV"WebSocketConnectionClosedException BaseExceptionrJrarer r])r, raw_messageresponse_messagermbe channel_queues r-rZz$GatewayKernelClient._route_responsesOs! M,,**666"11668 "#.tK/@#A *95++777$$W-889IJ,,$##///!1188:M59M 2; :;;;   M))  #EbT!KL MsBCD-0D-8+D((D-)TTTTT)r}r~rr allow_stdinrr8rYr]rrbrergrirkrZrrs@r-r=r=s"K33))$$''''''"!%F(*######  %%rs# ++@476+EE?? >!:l%";l%^K$0K$\N.R>.R>j./X/5X/v\v<+v