Notebook/IPython hooks

Pure IPython facilities to help us inspect, control, and modify cell outputs.

source

cellspec_from

 cellspec_from (s)
nburi, cellid = cellspec_from('/a/b/d#Y114sZmlsZQ==')
nburi, cellid
('/a/b/d', 'Y114sZmlsZQ==')

VSCode cell id format.

NB update

__nb__ = NB()
nb = NB()
nb.update(cell_id='abc', source='print(1)')  # type: ignore
test_eq(nb[0].source, 'print(1)')
def _update_cell(cell_id, source=None, exec_result=None):
    upd = {'execution_count': exec_result.execution_count} if exec_result else {}
    if source is not None: upd['source'] = source
    __nb__.update(cell_id, **upd, outputs=_get_outputs(exec_result))  # type: ignore

CellExecInfo

IPython cell execution info and __nb__ updater.

Simple IPython event callback that captures cell id and source code of last run cell.

NOTE: in this notebook, __nb__ is NOT a valid notebook state yet. It doesn’t reflect markdown cells or deleted cells or cells order. For a valid, updated in real-time, nbformat compliant notebook state (NB), see 21_nb_state.ipynb.

source

get_csi

 get_csi (start=False)

source

get_lastinfo

 get_lastinfo ()

source

get_info

 get_info ()

source

CellExecInfo

 CellExecInfo (start=False)

Initialize self. See help(type(self)) for accurate signature.

try: csi.stop()  # type: ignore
except Exception: pass

csi = get_csi(True)
test_eq(__cellinfo__, {})
test_eq(__lastcellinfo__, None)
show(DetailsJSON(__cellinfo__))
test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__")
show(DetailsJSON(__nb__[0]))
17
summary
  • source: show(DetailsJSON(__cellinfo__)) test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__") show(DetailsJSON(__nb__[0])) 17
  • cell_id: X35sZmlsZQ==
    • exec_result
    • result: None
summary
  • idx: 0
  • source: show(DetailsJSON(__cellinfo__)) test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__") show(DetailsJSON(__nb__[0])) 17
  • id: X35sZmlsZQ==
  • cell_type: code
    • outputs 
    17
    show(DetailsJSON(__lastcellinfo__, openall=True))
show(__nb__[__lastcellinfo__.cell_id])
    summary
    • source: show(DetailsJSON(__cellinfo__)) test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__") show(DetailsJSON(__nb__[0])) 17
    • cell_id: X35sZmlsZQ==
      • exec_result
      • result: 17
      • execution_count: 22
      • error_before_exec: None
      • error_in_exec: None
    summary
    • idx: 0
    • source: show(DetailsJSON(__cellinfo__)) test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__") show(DetailsJSON(__nb__[0])) 17
    • id: X35sZmlsZQ==
    • cell_type: code
      • outputs
        0
        • output_type: execute_result
        • execution_count: 22
          • data
          • text/plain: 17
          metadata
      • execution_count: 22

      __cellinfo__ stores information about current cell execution, interactiveshell.ExecutionInfo. __cellinfo__.exec_result stores the result of the cell execution, interactiveshell.ExecutionResult, only valid after the cell run.

      (ri := random.randint(0, sys.maxsize))
      540056390924159495
      show(DetailsJSON(__cellinfo__))
show(DetailsJSON(__lastcellinfo__, openall=True))
show(__nb__[__lastcellinfo__.cell_id])
test_eq(__lastcellinfo__.source, "(ri := random.randint(0, sys.maxsize))")
      summary
      • source: show(DetailsJSON(__cellinfo__)) show(DetailsJSON(__lastcellinfo__, openall=True)) show(__nb__[__lastcellinfo__.cell_id]) test_eq(__lastcelli…
      • cell_id: X42sZmlsZQ==
        • exec_result
        • result: None
      summary
      • source: (ri := random.randint(0, sys.maxsize))
      • cell_id: X41sZmlsZQ==
        • exec_result
        • result: 540056390924159495
        • execution_count: 24
        • error_before_exec: None
        • error_in_exec: None
      summary
      • idx: 2
      • source: (ri := random.randint(0, sys.maxsize))
      • id: X41sZmlsZQ==
      • cell_type: code
        • outputs
          0
          • output_type: execute_result
          • execution_count: 24
            • data
            • text/plain: 540056390924159495
            metadata
        • execution_count: 24 
        print(__cellinfo__.cell_id)
test_eq(__cellinfo__.source[:27], "print(__cellinfo__.cell_id)")
test_eq(In[-1][:27], "print(__cellinfo__.cell_id)")
(exe_cnt := len(In)-1)
        X43sZmlsZQ==
        26
        print(_, __lastcellinfo__.exec_result.result)
test_eq(_, __lastcellinfo__.exec_result.result)
test_eq(__lastcellinfo__.exec_result.result, Out[exe_cnt])
        26 26
        display('abc')
        'abc'
        print(_, __lastcellinfo__.exec_result.result)
test_eq(None, __lastcellinfo__.exec_result.result)
        26 None
        print('13')
23
        13
        23
        print(_, __lastcellinfo__.exec_result.result)
test_eq(23, __lastcellinfo__.exec_result.result)
        23 23
        csi.stop()
test_is(__cellinfo__, None)
test_is(__lastcellinfo__, None)

        __cellinfo__.source corresponds to In[-1] or _ih[-1].
        __lastcellinfo__.exec_result.result corresponds to _ or Out[__lastcellinfo__.exec_result.execution_count].

        But we don’t want the result of the last cell, we want the result of the current cell. For that, keep reading.

        Caveat: the front-end is not required to send the cell id. See nbformat Cell ids, run_cell.
        VSCode reports nbformat of newly created notebook as 4.4, but it does send the cell id, though not well formed.
        nbclassic does not set the cell ID even if the reported nbformat version is 4.5. Bridget creates one in this case.

        for c in __nb__.cells: display(c)
        NBCell@0
        • idx: 0
        • source: show(DetailsJSON(__cellinfo__)) test_eq(__cellinfo__.source[:29], "show(DetailsJSON(__cellinfo__") show(DetailsJSON(__nb__[0])) 17
        • id: X35sZmlsZQ==
        • cell_type: code
          • outputs
            0
            • output_type: execute_result
            • execution_count: 22
              • data
              • text/plain: 17
            • metadata: {}
        • execution_count: 22
        NBCell@1
        • idx: 1
        • source: show(DetailsJSON(__lastcellinfo__, openall=True)) show(__nb__[__lastcellinfo__.cell_id])
        • id: X36sZmlsZQ==
        • cell_type: code
          • outputs
          • execution_count: 23
          NBCell@2
          • idx: 2
          • source: (ri := random.randint(0, sys.maxsize))
          • id: X41sZmlsZQ==
          • cell_type: code
            • outputs
              0
              • output_type: execute_result
              • execution_count: 24
                • data
                • text/plain: 540056390924159495
              • metadata: {}
          • execution_count: 24
          NBCell@3
          • idx: 3
          • source: show(DetailsJSON(__cellinfo__)) show(DetailsJSON(__lastcellinfo__, openall=True)) show(__nb__[__lastcellinfo__.cell_id]) test_eq(__lastcelli…
          • id: X42sZmlsZQ==
          • cell_type: code
            • outputs
            • execution_count: 25
            NBCell@4
            • idx: 4
            • source: print(__cellinfo__.cell_id) test_eq(__cellinfo__.source[:27], "print(__cellinfo__.cell_id)") test_eq(In[-1][:27], "print(__cellinfo__.cell_i…
            • id: X43sZmlsZQ==
            • cell_type: code
              • outputs
                0
                • output_type: execute_result
                • execution_count: 26
                  • data
                  • text/plain: 26
                • metadata: {}
            • execution_count: 26
            NBCell@5
            • idx: 5
            • source: print(_, __lastcellinfo__.exec_result.result) test_eq(_, __lastcellinfo__.exec_result.result) test_eq(__lastcellinfo__.exec_result.result, O…
            • id: X44sZmlsZQ==
            • cell_type: code
              • outputs
              • execution_count: 27
              NBCell@6
              • idx: 6
              • source: display('abc')
              • id: X45sZmlsZQ==
              • cell_type: code
                • outputs
                • execution_count: 28
                NBCell@7
                • idx: 7
                • source: print(_, __lastcellinfo__.exec_result.result) test_eq(None, __lastcellinfo__.exec_result.result)
                • id: X46sZmlsZQ==
                • cell_type: code
                  • outputs
                  • execution_count: 29
                  NBCell@8
                  • idx: 8
                  • source: print('13') 23
                  • id: X50sZmlsZQ==
                  • cell_type: code
                    • outputs
                      0
                      • output_type: execute_result
                      • execution_count: 30
                        • data
                        • text/plain: 23
                      • metadata: {}
                  • execution_count: 30
                  NBCell@9
                  • idx: 9
                  • source: print(_, __lastcellinfo__.exec_result.result) test_eq(23, __lastcellinfo__.exec_result.result)
                  • id: X51sZmlsZQ==
                  • cell_type: code
                    • outputs
                    • execution_count: 31
                    NBCell@10
                    • idx: 10
                    • source: csi.stop() test_is(__cellinfo__, None) test_is(__lastcellinfo__, None)
                    • id: X52sZmlsZQ==
                    • cell_type: code
                      • outputs

                      autoid (not used)

                      Note: autoid here is different from fasthtml fh_cfg['auto_id'] option. Here we’re trying to automatically set the id attribute of the wrapper element of a cell output in the front-end. If we can do so, we’ll be able to target especific cell outputs.

                      __autoid_scr = '''
// debugger;
me().attribute('id', 'output-{0}').classAdd('bridge');
setTimeout(el => {{ el.remove(); }}, 100, me('#{0}'))
'''
def autoid(idx=None):
    idx = idx or new_id()
    return Script(__autoid_scr.format(idx), id=idx), idx
                      scr, idx = autoid()
dhdl = DisplayHandle(idx)
dhdl.display(HTML('bbbb'+to_xml(scr)))
                      cccc

                      Inspect “bbbb” output of previous cell, the parent element should have class “bridge” and id.

                      dhdl.update(HTML('cccc'+to_xml(scr)))

                      VSCode wipe out the element when updating the cell. We need to send again the autoid.

                      DisplayId

                      An attempt to provide a IPython display wrapper that automatically handles the display ID to allow us to target especific cells. Not working, for now it’s essentially just IPython display.

                      class DisplayId(DisplayHandle):
    def __init__(self, display_id=None):
        super().__init__(display_id or new_id())
        self._contents = None
        self._sc = to_xml(autoid(self.display_id)[0]) if bridge_cfg.auto_id else ''

    def display(self, obj='', **kwargs):
        self._contents = str(obj)
        IDISPLAY(HTML(self._contents + self._sc), display_id=self.display_id, **kwargs)

    def update(self, obj='', **kwargs):
        kwargs['update'] = True
        self.display(obj, **kwargs)

    def contents(self): return self._contents
                      dhdl = DisplayId()
dhdl.display('dddd')
test_eq(dhdl.contents(), 'dddd')
                      dhdl.update('eeee')
test_eq(dhdl.contents(), 'eeee')
                      dhdl.update()

                      Bridged: IPython display_pub hook

                      display_pub hook for display_id and brd-mark
                      Tag cell outputs with bridge metadata to target them.

                      In particular, it will transform every display message to transient if the message has a session metadata id (brd_did). It will set the display_id of each output with that same brd_id value. For HTML display objects, it also marks the DOM parent element in the front-end. With this (session unique) tag, we can easily address specific outputs from Python.

                      This will be handy to target specific cell outputs when we can capture the notebook state down the road.

                      NOTE: display_pub hooks are thread dependent. Here we assume we only set the hook from the main thread.

                      source

                      get_bridged

                       get_bridged (start=False)

                      source

                      Bridged

                       Bridged (start=False)

                      Augment display messages with bridge stuff.

                      try: 
    get_csi().stop()
    brdd.stop()  # type: ignore
except Exception: pass

__nb__ = NB()  # type: ignore

bridge_cfg.auto_id = False
brdd = get_bridged(True)

                      set display_id with display object metadata

                      {
    ...,
    'msg_type': 'display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>', 
            'text/html': "<div>I'm marked!... MAAARKED!!</div>"
        },
        'metadata': {'text/html': {'brd_did': 'b8b568b9a-c02e1576-c3a3c120-167cedda'}},
        'transient': {}
    },
    'metadata': {}
}

{
    ...,
    'msg_type': 'display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>',
            'text/html': '<div>I\'m marked!... MAAARKED!!</div><brd-mark id="b8b568b9a-c02e1576-c3a3c120-167cedda"></brd-mark>'
        },
        'metadata': {'text/html': {'brd_did': 'b8b568b9a-c02e1576-c3a3c120-167cedda'}},
        'transient': {'display_id': 'b8b568b9a-c02e1576-c3a3c120-167cedda'}
    },
    'metadata': {}
}
                      did = new_id()
print(f"{did=}")
display(HTML("<div>I'm marked!... MAAARKED!!</div>", metadata={'brd_did': did}))
info = __cellinfo__; cell1  = __nb__[info.cell_id]; cell1_copy = cell1.copy()
test_eq(cell1.dids, [did])  # NOTE: this is only valid after the `display(...)` call in above line
21
                      did='b595555a9-c867f02c-47d4b361-005ec7c4'
                      I'm doomed!... DOOOOOMED!!
                      21
                      display(cell1_copy)
cell1
                      NBCell@0
                      • idx: 0
                      • source: did = new_id() print(f"{did=}") display(HTML("<div>I'm marked!... MAAARKED!!</div>", metadata={'brd_did': did})) info = __cellinfo__; cell1 …
                      • id: Y120sZmlsZQ==
                      • cell_type: code
                        • outputs
                          0
                          • output_type: display_data
                            • data
                            • text/plain: <IPython.core.display.HTML object>
                            • text/html: <div>I'm marked!... MAAARKED!!</div> <brd-mark id="b595555a9-c867f02c-47d4b361-005ec7c4"></brd-mark>
                          • metadata: {'brd_did': 'b595555a9-c867f02c-47d4b361-005ec7c4'}
                      NBCell@0
                      • idx: 0
                      • source: did = new_id() print(f"{did=}") display(HTML("<div>I'm marked!... MAAARKED!!</div>", metadata={'brd_did': did})) info = __cellinfo__; cell1 …
                      • id: Y120sZmlsZQ==
                      • cell_type: code
                        • outputs
                          0
                          • output_type: display_data
                            • data
                            • text/plain: <IPython.core.display.HTML object>
                            • text/html: <div>I'm marked!... MAAARKED!!</div> <brd-mark id="b595555a9-c867f02c-47d4b361-005ec7c4"></brd-mark>
                          • metadata: {'brd_did': 'b595555a9-c867f02c-47d4b361-005ec7c4'}
                          1
                          • output_type: execute_result
                          • execution_count: 50
                            • data
                            • text/plain: 21
                          • metadata: {}
                      • execution_count: 50

                      At cell runtime, current cell can be accesed as __nb__[__cellinfo__.cell_id]; after cell execution, it can be accesed as __nb__[__lastcellinfo__.cell_id].

                      Note NBCell instance lifecycle: - Before code execution: an instance is created with source and id - After display statement: the instance is updated with display_data output - After cell run: the instance is updated with execute_result output

                      test_is('<div>I\'m marked!... MAAARKED!!</div>' in cell1.outputs[0]['data']['text/html'], True)
dh = brdd.dhs[-1]
test_eq(dh.display_id, did)

                      For convenience, Bridged stores in dhs the last display handles used.

                      {
    ...,
    'msg_type': 'update_display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>', 
            'text/html': "<div>I'm doomed!... DOOOOOMED!!</div>"},
        'metadata': {},
        'transient': {'display_id': 'b5c00d851-9da4a95e-36473c24-3d04534d'}
    },
    'metadata': {}
}

{
    ...,
    'msg_type': 'update_display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>',
            'text/html': '<div>I\'m doomed!... DOOOOOMED!!</div><brd-mark id="b5c00d851-9da4a95e-36473c24-3d04534d"></brd-mark>'
        },
        'metadata': {},
        'transient': {'display_id': 'b5c00d851-9da4a95e-36473c24-3d04534d'}
    },
    'metadata': {}
}
                      dh.update(HTML("<div>I'm doomed!... DOOOOOMED!!</div>"))
cell2 = __nb__[__cellinfo__.cell_id]
test_is(hasattr(cell2, 'did'), False)
htmls = vals_at(cell1, 'outputs.*.data.text/html', True)
test_is(any('MAAARKED' in v for v in htmls), False)
test_is(any('DOOOOOMED' in v for v in htmls), True)
                      display(cell1)
cell2
                      NBCell@0
                      • idx: 0
                      • source: did = new_id() print(f"{did=}") display(HTML("<div>I'm marked!... MAAARKED!!</div>", metadata={'brd_did': did})) info = __cellinfo__; cell1 …
                      • id: Y120sZmlsZQ==
                      • cell_type: code
                        • outputs
                          0
                          • output_type: display_data
                            • data
                            • text/plain: <IPython.core.display.HTML object>
                            • text/html: <div>I'm doomed!... DOOOOOMED!!</div> <brd-mark id="b595555a9-c867f02c-47d4b361-005ec7c4"></brd-mark>
                          • metadata: {'brd_did': 'b595555a9-c867f02c-47d4b361-005ec7c4'}
                          1
                          • output_type: execute_result
                          • execution_count: 50
                            • data
                            • text/plain: 21
                          • metadata: {}
                      • execution_count: 50
                      NBCell@3
                      • idx: 3
                      • source: dh.update(HTML("<div>I'm doomed!... DOOOOOMED!!</div>")) cell2 = __nb__[__cellinfo__.cell_id] test_is(hasattr(cell2, 'did'), False) htmls = …
                      • id: Y126sZmlsZQ==
                      • cell_type: code
                        • outputs
                        • execution_count: 53

                        set display_id with display call metadata

                        i.e., display(..., display_id=True) or display(..., display_id="...")

                        dh = display(HTML("<div>I'm marked!... MAAARKED!!</div>"), display_id=True)
cell = __nb__[__cellinfo__.cell_id]
test_eq(at(cell, 'outputs.0.metadata.brd_did'), brdd.dh.display_id)  # type: ignore
cell
                        I'm doomed!... DOOOOOMED!!
                        NBCell@5
                        • idx: 5
                        • source: dh = display(HTML("<div>I'm marked!... MAAARKED!!</div>"), display_id=True) cell = __nb__[__cellinfo__.cell_id] test_eq(at(cell, 'outputs.0.…
                        • id: Y132sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>I'm marked!... MAAARKED!!</div> <brd-mark id="e2ffaa8dc80ef2623bc8eb433633f96a"></brd-mark>
                            • metadata: {'brd_did': 'e2ffaa8dc80ef2623bc8eb433633f96a'} 
                        {
    ...,
    'msg_type': 'display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>', 
            'text/html': "<div>I'm marked!... MAAARKED!!</div>"
        },
        'metadata': {},
        'transient': {'display_id': '2307db4acc4fda0ba305ffdda518748a'}
    },
    'metadata': {}
}

{
    ...,
    'msg_type': 'display_data',
.    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>',
            'text/html': '<div>I\'m marked!... MAAARKED!!</div><brd-mark id="2307db4acc4fda0ba305ffdda518748a"></brd-mark>'
        },
        'metadata': {'brd_did': '2307db4acc4fda0ba305ffdda518748a'},
        'transient': {'display_id': '2307db4acc4fda0ba305ffdda518748a'}
    },
    'metadata': {}
}
                        brdd.dh.update(HTML("<div>I'm doomed!... DOOOOOMED!!</div>"))  # type: ignore
cell
                        NBCell@5
                        • idx: 5
                        • source: dh = display(HTML("<div>I'm marked!... MAAARKED!!</div>"), display_id=True) cell = __nb__[__cellinfo__.cell_id] test_eq(at(cell, 'outputs.0.…
                        • id: Y132sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>I'm doomed!... DOOOOOMED!!</div> <brd-mark id="e2ffaa8dc80ef2623bc8eb433633f96a"></brd-mark>
                            • metadata: {'brd_did': 'e2ffaa8dc80ef2623bc8eb433633f96a'}
                            1
                            • output_type: execute_result
                            • execution_count: 55
                              • data
                              • text/plain: {'idx': 5, 'source': 'dh = display(HTML("<div>I\'m marked!... MAAARKED!!</div>"), display_id=True)\ncell = __nb__[__cellinfo__.cell_id]\nte…
                              • text/html: <style>details ul { list-style-type:none; list-style-position: outside; padding-inline-start: 22px; margin: 0; } details .string { color: #2…
                              • text/markdown: > code ```json { 'idx': 5, 'source': 'dh = display(HTML("<div>I\'m marked!... MAAARKED!!</div>"), display_id=True)\ncell = __nb__[__cel…
                            • metadata: {}
                        • execution_count: 55 
                        {
    ...,
    'msg_type': 'update_display_data',
    'content': {
        'data': {'text/plain': '<IPython.core.display.HTML object>', 'text/html': "<div>I'm doomed!... DOOOOOMED!!</div>"},
        'metadata': {},
        'transient': {'display_id': 'c3d21633d341d2463f13ef40730e8c4a'}
    },
    'metadata': {}
}

{
    ...,
    'msg_type': 'update_display_data',
    'content': {
        'data': {
            'text/plain': '<IPython.core.display.HTML object>',
            'text/html': '<div>I\'m doomed!... DOOOOOMED!!</div><brd-mark id="c3d21633d341d2463f13ef40730e8c4a"></brd-mark>'
        },
        'metadata': {},
        'transient': {'display_id': 'c3d21633d341d2463f13ef40730e8c4a'}
    },
    'metadata': {}
}
                        display('aaaa', display_id=new_id());
                        bbbb 
                        brdd.dh.update(HTML("bbbb"))  # type: ignore
                        display('cccc', metadata={'bridge': {'brd_did': new_id()}})
                        dddd 
                        brdd.dh.update(HTML("dddd"))  # type: ignore
                        display('eeee', metadata={'test/plain': {'brd_did': new_id()}})
test_eq(__nb__[__cellinfo__.cell_id].outputs[0].data['text/plain'], "'eeee'")

                        ffff

                        brdd.dh.update(Markdown("ffff"))  # type: ignore
test_eq(__nb__[__cellinfo__.cell_id].outputs, [])

                        Multi objects display

                        dh = display(
    HTML("<div>Multi 1</div>"), 
    HTML("<div>Multi 2</div>"), 
    display_id=True)
                        Multi 1
                        Multi 3
                        (cell := __nb__[__lastcellinfo__.cell_id])
                        NBCell@13
                        • idx: 13
                        • source: dh = display( HTML("<div>Multi 1</div>"), HTML("<div>Multi 2</div>"), display_id=True)
                        • id: Y150sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>Multi 1</div> <brd-mark id="6ff9be5be58ae578558d4b803e5a6b9e"></brd-mark>
                            • metadata: {'brd_did': '6ff9be5be58ae578558d4b803e5a6b9e'}
                            1
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>Multi 2</div> <brd-mark id="6ff9be5be58ae578558d4b803e5a6b9e"></brd-mark>
                            • metadata: {'brd_did': '6ff9be5be58ae578558d4b803e5a6b9e'}
                        • execution_count: 63 
                        if dh: dh.update(HTML("<div>Multi 3</div>"))
cell
                        NBCell@13
                        • idx: 13
                        • source: dh = display( HTML("<div>Multi 1</div>"), HTML("<div>Multi 2</div>"), display_id=True)
                        • id: Y150sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>Multi 1</div> <brd-mark id="6ff9be5be58ae578558d4b803e5a6b9e"></brd-mark>
                            • metadata: {'brd_did': '6ff9be5be58ae578558d4b803e5a6b9e'}
                            1
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: <div>Multi 3</div> <brd-mark id="6ff9be5be58ae578558d4b803e5a6b9e"></brd-mark>
                            • metadata: {'brd_did': '6ff9be5be58ae578558d4b803e5a6b9e'}
                        • execution_count: 63

                        When using transient display messages with the display function, multi objects display is ill-defined.

                        IPython display assign the same display_id to each object. The front-end however will handle it differently.

                        VSCode displays all objects but only consider transient the last one.
                        Lab/Notebook repeats the last object as many times as the number of objects sent.

                        display(
    HTML("<div>Multi 1</div>", metadata={'brd_did': (did1 := new_id())}), 
    HTML("<div>Multi 2</div>", metadata={'brd_did': (did2 := new_id())})
)
                        Multi 1
                        Multi 4
                        cell = __nb__[__lastcellinfo__.cell_id]
test_eq(cell.dids, [did1, did2])
                        dh1, dh2 = brdd.dhs[-2], brdd.dhs[-1]
test_eq(dh1.display_id, did1)
test_eq(dh2.display_id, did2)
dh1.update(HTML("<div>Multi 3</div>"))
time.sleep(0.01)
dh2.update(HTML("<div>Multi 4</div>"))

                        We can sidestep the issue by using specific Bridge metadata.

                        skip

                        display(Markdown("Skipped"), metadata={'bridge': {'skip': True}})
display("Me too", JSON({"And me": True}), metadata=skip())

                        Skipped

                        'Me too'
                        <IPython.core.display.JSON object>
                        __nb__[__lastcellinfo__.cell_id]
                        NBCell@19
                        • idx: 19
                        • source: display(Markdown("Skipped"), metadata={'bridge': {'skip': True}}) display("Me too", JSON({"And me": True}), metadata=skip())
                        • id: Y162sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.Markdown object>
                              • text/markdown: Skipped
                            • metadata: {'bridge': {'skip': True}}
                            1
                            • output_type: display_data
                              • data
                              • text/plain: 'Me too'
                            • metadata: {'bridge': {'skip': True}}
                            2
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.JSON object>
                              • application/json: {'And me': True}
                            • metadata: {'application/json': {'expanded': False, 'root': 'root'}, 'bridge': {'skip': True}}
                        • execution_count: 69

                        Skip tagging all display objects.

                        display(
    HTML("Skipped", metadata={'skip':True, 'brd_did':(did1 := new_id())}),
    HTML("Not me", metadata={'brd_did':(did2 := new_id())}),
)
                        Skipped
                        Not me 
                        cell = __nb__[__lastcellinfo__.cell_id]
test_eq(cell.dids, [did2])
cell
                        NBCell@21
                        • idx: 21
                        • source: display( HTML("Skipped", metadata={'skip':True, 'brd_did':(did1 := new_id())}), HTML("Not me", metadata={'brd_did':(did2 := new_id()…
                        • id: Y165sZmlsZQ==
                        • cell_type: code
                          • outputs
                            0
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: Skipped
                            • metadata: {'text/html': {'skip': True, 'brd_did': 'b19bbdaeb-1ae6ed40-ce525c18-23d81243'}}
                            1
                            • output_type: display_data
                              • data
                              • text/plain: <IPython.core.display.HTML object>
                              • text/html: Not me <brd-mark id="b55581abb-bb1b69e3-6f11a7e8-f4d20014"></brd-mark>
                            • metadata: {'brd_did': 'b55581abb-bb1b69e3-6f11a7e8-f4d20014'}
                        • execution_count: 71

                        Skip specific display object.

                        bridge_cfg.auto_id

                        if bridge_cfg.auto_id is True, there’s no need to use bridge metadata. Every supported displayed object (see _BRDD_MIMES) will receive an auto-generated display id.

                        Caveat: be aware that VSCode limits the number of transient display ids (1000 last time I checked); not Jupyter, I believe.

                        bridge_cfg.auto_id = True

display(HTML("<div>I'm auto-id'd--</div>"))
                        --as shown above.
                        brdd.dhs[-1].update(HTML("<div>--as shown above.</div>"))
                        display(Markdown(f"## did\n???"))
                        <IPython.core.display.JSON object>
                        dh = brdd.dhs[-1]
dh.update(HTML(f"<b>did</b>: {dh.display_id}"))
                        dh.update(JSON({'did': dh.display_id}))
                        dhs = []
for i in range(5): 
    display(f'{i=}')
    dhs.append(brdd.dhs[-1])
                        'i+1=1'
                        'i+1=2'
                        'i+1=3'
                        'i+1=4'
                        'i+1=5'
                        for i in range(5): 
    dhs[i].update(f'{i+1=}')
                        brdd.dhs.clear()
brdd.stop()
csi.stop()
bridge_cfg.auto_id = False
                        def show_msgs(brdd: Bridged):
    for msg in brdd.msgs.copy():
        d = msg.copy()
        # d['parent_header'] = {'...': '...'}
        # d['header'] = {'...': '...'}
        del d['parent_header'], d['header'], d['tracker'], d['msg_id']
        if not d['metadata']: del d['metadata']
        try: del d['content']['data']['text/plain']
        except: pass
        if h := d['content']['data'].get('text/html'): d['content']['data']['text/html'] = shorten(h, 'r', 120)
        cprint(d)

if DEBUG(): show_msgs(brdd)
                        for c in __nb__.cells: display(c)

                        OutputCapture

                        Bridget goal is to control (at least) all HTML output. Bridge can now set metadata of any display message, those that go through display_pub. Bridge captures all display, direct or FastHTML bridge.
                        But there’s other way to produce output that doesn’t follow the display_pub path: auto display of cell’s final expression. That goes through another code path, display_hook.
                        Here Bridget leverage IPython’s own capture mechanism to intercept cell results and redirect to display, a path that Bridge already control.

                        Bridge just captures cell outputs, not stdout/err (yet)

                        def _transform(lines):
    "Input transformer function"
    cpt = get_capturer()
    if not lines or cpt._capturing or cpt._debugging: return lines
    if DEBUG(): cpt._lines.append(lines)
    if lines[0].startswith('import debugpy'):
        cpt._debugging = True
        return lines
    elif lines[0].startswith('import debugpy;debugpy.listen('): return lines
    elif lines[0].startswith('import debugpy\ndebugpy.debug_this_thread()'): return lines
    elif lines[0].startswith('def __jupyter_exec_background__()'): return lines
    elif lines[0].startswith('import builtins') and lines[1].startswith('import ipykernel'): return lines
    elif lines[0].startswith('import os as _VSCODE_os') and lines[1].startswith('_VSCODE_fileList ='): return lines
    return ['get_capturer()(%r)\n' % ''.join(lines)]
_transform.has_side_effects = False


class OutputCapture:
    shell: InteractiveShell
    def __init__(self):
        super().__init__()
        self._active, self.shell = False, get_ipython()  # type: ignore
        if DEBUG(): self._captures = deque(maxlen=100); self._lines = deque(maxlen=100)
        self._capturing, self._debugging, self.run_outputs = False, False, []
        self.displayhook = CapturingDisplayHook(shell=self.shell, outputs=self.run_outputs)
    @property
    def active(self): return self._active
    def start(self):
        if self._active: return
        self._active = True
        self.shell.user_ns['get_capturer'] = get_capturer
        if DEBUG(): self._captures = deque(maxlen=100)
        # shell.input_transformer_manager.line_transforms.append(_transform)
        self.shell.input_transformers_post.append(_transform)
    def stop(self):
        if not self._active: return
        self._active = False
        # try: shell.input_transformer_manager.line_transforms.remove(_transform)
        try: self.shell.input_transformers_post.remove(_transform)
        except (ValueError, NameError): pass
    def __del__(self): self.stop()

    @contextmanager
    def _capture(self):
        self.run_outputs.clear()
        try: 
            save_display_hook, sys.displayhook = sys.displayhook, self.displayhook
            self._capturing = True
            yield CapturedIO(stdout=None, stderr=None, outputs=self.run_outputs)
        finally: 
            self._capturing = False
            sys.displayhook = save_display_hook
    def __call__(self, cell):
        info: AD = self.shell.user_ns.get('__cellinfo__')  # type: ignore
        with self._capture() as io:
            self.shell.run_cell(cell, cell_id=info.cell_id)
        if DEBUG(): self._captures.append([cell, io._outputs.copy()])
        if io._outputs: 
            assert len(io._outputs) <= 1, "Only one output is supported"
            info.exec_result.result = io._outputs[-1]
            display(io.outputs[-1], metadata={'bridge': {'captured': True}})
    
__capturer__ = None
def get_capturer(start:bool=False):
    global __capturer__
    get_csi(True)
    if __capturer__ is None: __capturer__ = OutputCapture()
    if start: __capturer__.start()
    return __capturer__
                        get_csi().stop()
get_bridged().stop()
get_capturer().stop()

bridge_cfg.auto_id = True

__nb__ = NB()  # type: ignore

brdd = get_bridged(True)

cptr = get_capturer(True)
                        1+3
                        I was 4, now I'm 44
                        info = __lastcellinfo__
test_eq(__nb__[__lastcellinfo__.cell_id].outputs[0].data, {'text/plain': '4'})
test_eq(len(brdd.dhs), 1)
show(DetailsJSON(__lastcellinfo__, openall=True))
                        summary
                        • source: 1+3
                        • cell_id: Y222sZmlsZQ==
                          • exec_result
                            result
                              data
                              • text/plain: 4
                              metadata
                            • execution_count: 85
                            • error_before_exec: None
                            • error_in_exec: None

                          __cellinfo__.result has a valid value only after the display(...) call. The cell with 1+3 captures the output and then displays it with display(...). That occurs after the cell is executed. So, __cellinfo__.result is None during the cell execution. It’s only possible to get the output after the cell has run.

                          HTML('<div>asdf</div>')
                          I was asdf, now I'm qwer
                          if DEBUG(): 
    output_data = get_capturer()._captures[-1][1][0]['data']
    test_eq(output_data['text/plain'], '<IPython.core.display.HTML object>')
    html = output_data['text/html']
    test_eq(f'<brd-mark id="{brdd.dh.display_id}"' in html, True)  # type: ignore
test_eq(len(brdd.dhs), 3)
                          print(10)
17
                          10
                          I was 17, now I'm 177
                          1/0
                          ---------------------------------------------------------------------------
ZeroDivisionError                         Traceback (most recent call last)
Cell In[90], line 1
----> 1 1/0

ZeroDivisionError: division by zero
                          get_bridged() .stop()
get_csi().stop()
get_capturer().stop()
                          brdd.dhs[0].update(HTML("<div>I was 4, now I'm 44</div>"))
                          brdd.dhs[2].update(HTML("<div>I was asdf, now I'm qwer</div>"))
                          brdd.dh.update(HTML("<div>I was 17, now I'm 177</div>"))  # type: ignore
                          def show_captures():
    cc = get_capturer()._captures
    for c in cc:
        cprint(c)
        print('---------')

show_captures()
                          ['1+3\n', [{'data': {'text/plain': '4'}, 'metadata': {}}]]

                          ---------
                          [
    "info = __lastcellinfo__\ntest_eq(__nb__[__lastcellinfo__.cell_id].outputs[0].data, {'text/plain': '4'})\ntest_eq(len(brdd.dhs), 
1)\nshow(DetailsJSON(__lastcellinfo__, openall=True))\n",
    []
]

                          ---------
                          [
    "HTML('<div>asdf</div>')\n",
    [
        {
            'data': {
                'text/plain': '<IPython.core.display.HTML object>',
                'text/html': '<div>asdf</div>\n<brd-mark id="b3c829493-53426d78-6c9050ad-6e20e069"></brd-mark>'
            },
            'metadata': {}
        }
    ]
]

                          ---------
                          [
    'if DEBUG(): \n    output_data = get_capturer()._captures[-1][1][0][\'data\']\n    test_eq(output_data[\'text/plain\'], 
\'<IPython.core.display.HTML object>\')\n    html = output_data[\'text/html\']\n    test_eq(f\'<brd-mark id="{brdd.dh.display_id}"\' in 
html, True)  # type: ignore\ntest_eq(len(brdd.dhs), 3)\n',
    []
]

                          ---------
                          ['print(10)\n17\n', [{'data': {'text/plain': '17'}, 'metadata': {}}]]

                          ---------
                          ['1/0\n', []]

                          ---------
                          ['get_bridged() .stop()\nget_csi().stop()\nget_capturer().stop()\n', []]

                          ---------
                          def show_lines():
    cc = get_capturer()._lines
    for c in cc:
        cprint(shorten(c, 'r', 140))
        print('---------')

show_lines()
                          if DEBUG(): show_msgs(brdd)  # type: ignore
                          {
    'msg_type': 'display_data',
    'content': {
        'data': {},
        'metadata': {'bridge': {'captured': True}},
        'transient': {'display_id': 'b34bd6418-582983a0-eaaadd4f-12e98d9e'}
    }
}

                          {
    'msg_type': 'display_data',
    'content': {
        'data': {
            'text/html': '<style>details ul { list-style-type:none; list-style-position: outside; padding-inline-start: 22px; margin: 0; } 
details…'
        },
        'metadata': {},
        'transient': {'display_id': 'bc4743314-1703b8af-012c6759-46fba1e2'}
    }
}

                          {
    'msg_type': 'display_data',
    'content': {
        'data': {'text/html': '<div>asdf</div>\n<brd-mark id="b3c829493-53426d78-6c9050ad-6e20e069"></brd-mark>'},
        'metadata': {'bridge': {'captured': True}},
        'transient': {'display_id': 'b3c829493-53426d78-6c9050ad-6e20e069'}
    }
}

                          {
    'msg_type': 'display_data',
    'content': {
        'data': {},
        'metadata': {'bridge': {'captured': True}},
        'transient': {'display_id': 'b4e7c9b47-1eeb4595-2b85a414-25b2e188'}
    }
}

                          CaptureTransformer

                          Output capture with AST hooks.

                          OutputCapture works correctly, but conflicts with the debugger abounds as it alters the source code. Fortunately, IPython has another much more powerful hook mechanism, ast_transformers, that is cleaner and more flexible.

                          def _ast_process_result(result):
    if result is not None:
        if not (shell := get_ipython()): return result
        info = shell.user_ns['__cellinfo__']
        if result is not info: info.exec_result.result = result
        display(result, metadata={'bridge': {'captured': True}})

                          source

                          get_capturer

                           get_capturer (start=False)

                          source

                          CaptureTransformer

                           CaptureTransformer (mode='direct')

                          *A :class:NodeVisitor subclass that walks the abstract syntax tree and allows modification of nodes.

                          The NodeTransformer will walk the AST and use the return value of the visitor methods to replace or remove the old node. If the return value of the visitor method is None, the node will be removed from its location, otherwise it is replaced with the return value. The return value may be the original node in which case no replacement takes place.

                          Here is an example transformer that rewrites all occurrences of name lookups (foo) to data['foo']::

                          class RewriteName(NodeTransformer):

                             def visit_Name(self, node):
       return Subscript(
           value=Name(id='data', ctx=Load()),
           slice=Constant(value=node.id),
           ctx=node.ctx
       )

                          Keep in mind that if the node you’re operating on has child nodes you must either transform the child nodes yourself or call the :meth:generic_visit method for the node first.

                          For nodes that were part of a collection of statements (that applies to all statement nodes), the visitor may also return a list of nodes rather than just a single node.

                          Usually you use the transformer like this::

                          node = YourTransformer().visit(node)*

                          get_csi().stop()
get_bridged().stop()
get_capturer().stop()

bridge_cfg.auto_id = True

__nb__ = NB()  # type: ignore

brdd = get_bridged()

cptr = get_capturer(True)
test_eq(get_ipython().ast_transformers, [cptr])  # type: ignore
                          node = cptr.visit_Module(ast.parse('''
x = 10
y = 20
x + y
'''))

test_eq(ast.unparse(node), 'x = 10\ny = 20\n_ast_process_result(x + y)')
                          x = 10
y = 20
x + y
                          30
                          test_is(_ != 30, True)
# test_eq(__lastcellinfo__.exec_result.result, 30)
show(DetailsJSON(__lastcellinfo__, summary='__lastcellinfo__', openall=True))
__nb__[__lastcellinfo__.cell_id]
                          __lastcellinfo__
                          • source: x = 10 y = 20 x + y
                          • cell_id: Y251sZmlsZQ==
                            • exec_result
                            • result: 30
                            • execution_count: 102
                            • error_before_exec: None
                            • error_in_exec: None
                          NBCell@1
                          • idx: 1
                          • source: x = 10 y = 20 x + y
                          • id: Y251sZmlsZQ==
                          • cell_type: code
                            • outputs
                              0
                              • output_type: display_data
                                • data
                                • text/plain: 30
                              • metadata: {'bridge': {'captured': True}, 'brd_did': 'be2adf707-956b7807-c1810422-23297835'}
                          • execution_count: 102

                          Note that CaptureTranformer changes the semantics of IPython code execution because is effectively disabling the Output caching system as it’s intercepting all cell outputs. If you have any use for _|_<n>|_oh|Out variables, CaptureTranformer has the same effect as setting InteractiveShell.cache_size to 0.

                          During cell execution, IPython replaces sys.displayhook with a custom DisplayHook instance responsible for displaying the result of the cell execution (among many other things). Bridget now handles cell outputs (and does what displayhook did before to show the cell result). The shell displayhook then always receives a result of None. Bridget must replicate some (not sure what to do about the shell.history_manager) of the functionality of displayhook to ensure that output cache variables are updated correctly (see __call__).

                          class TestD:
    def _ipython_display_(self):
        # dhdl = DisplayId()
        # dhdl.display(self.text)
        IDISPLAY(HTML('from _ipython_display_'))

TestD()
                          from _ipython_display_ 
                          __nb__[__lastcellinfo__.cell_id]
                          NBCell@4
                          • idx: 4
                          • source: class TestD: def _ipython_display_(self): # dhdl = DisplayId() # dhdl.display(self.text) IDISPLAY(HTML('from _ip…
                          • id: Y255sZmlsZQ==
                          • cell_type: code
                            • outputs
                              0
                              • output_type: display_data
                                • data
                                • text/plain: <IPython.core.display.HTML object>
                                • text/html: from _ipython_display_ <brd-mark id="b91d17647-da0db986-c9ab9be4-fb83af67"></brd-mark>
                              • metadata: {'brd_did': 'b91d17647-da0db986-c9ab9be4-fb83af67'}
                          • execution_count: 105 
                          bridge_cfg.auto_show = False
                          Div('Hey, Foo!')
                          <div>Hey, Foo!</div>
                          __nb__[__lastcellinfo__.cell_id]
                          NBCell@7
                          • idx: 7
                          • source: Div('Hey, Foo!')
                          • id: Y261sZmlsZQ==
                          • cell_type: code
                            • outputs
                              0
                              • output_type: display_data
                                • data
                                • text/plain: div(('Hey, Foo!',),{})
                                • text/markdown: ```html <div>Hey, Foo!</div> ```
                              • metadata: {'bridge': {'captured': True}, 'brd_did': 'bb5b04995-eaac4b3d-83f7e6db-c103ca68'}
                              1
                              • output_type: execute_result
                              • execution_count: 108
                                • data
                                • text/plain: div(('Hey, Foo!',),{})
                                • text/markdown: ```html <div>Hey, Foo!</div> ```
                              • metadata: {}
                          • execution_count: 108

                          stop_hooks

                          get_nb_from_hooks

                          _BRDD_MIMES
                          {'application/javascript',
 'application/json',
 'application/pdf',
 'image/jpeg',
 'image/png',
 'image/svg+xml',
 'text/html',
 'text/latex',
 'text/markdown',
 'text/plain'}

                          What does this module do?

                          • Receives IPython’s events to record cell info (what the kernel can possibly know before and after the cell run).
                          • Hooks into the display system to convert all calls into transient calls with a display_id connected to the cell_id.
                          • Transforms cell code with a custom AST transformer to redirect cell result to the display system.

                          This way, Bridget effectively knows what each cell output is (except stdout/err for now) and how to reference and modify it using standard IPython features.

                          Is this enough to make Bridget a functional notebook editor? Not really, we need real-time updates of the notebook structure (the notebook state, what would be saved to disk as .ipynb) to be able to navigate the notebook.

                          That unfortunately requires to navigate the procellous waters of widgets and extensions. Good ol’ IPython and Jupyter are not designed to give the kernel knowledge about the notebook state in real time. We pythonistas deluded ourselves into thinking Jupiter is all about us, but a Jupyter notebook is really a JavaScript application that controls everything, including the model and the view. The kernel is a second class citizen that knows next to nothing about or even what is a notebook.

                          We’re going to fix that next.