Parsing a Mardown python API to provide a context to an AI

Plan

We have a python package API documentation that is formatted as a Markdown document, using griffe2md. We want to use a package called toolslm to massage this information into python dictionaries so that an AI can exploit this information. The toolslm package has a few buit-in behaviors entirely sensible for some other inputs, that create artefacts in its outputs that are not ideal for the Markdown document produced by griffe2md. This document is an attempt to modify or augment toolslm to maintain python function names and more intuitive hierarchical section names for the API documentation.

Installing `toolslm` from source

We do need to force the installation of the toolslm from source (somehow found that I needed to use --force-reinstall, and make sure I restart solveit to get the right version), as it includes a fix not yet released in the toolslm on pypi.

# !pip install --force-reinstall "git+https://github.com/AnswerDotAI/toolslm"

!pip freeze | grep toolslm

toolslm @ git+https://github.com/AnswerDotAI/toolslm@7c959bff55a60479e412385b3933ba8dfcbb4b40

/usr/local/lib/python3.11/site-packages/lancedb/__init__.py:220: UserWarning: lance is not fork-safe. If you are using multiprocessing, use spawn instead.
  warnings.warn(

from bs4 import BeautifulSoup as bs
fn = "./static/swift2_api.md"
with open(fn, 'r') as file:
    content = file.read()
text = bs(content, 'html.parser').get_text()

print(text[:10000])

## swift2

Python interface to the SWIFT2 streamflow simulation and forecasting libraries.

**Modules:**

- [**classes**](#swift2.classes) –
- [**common**](#swift2.common) –
- [**const**](#swift2.const) –
- [**doc_helper**](#swift2.doc_helper) –
- [**helpers**](#swift2.helpers) –
- [**internal**](#swift2.internal) –
- [**model_definitions**](#swift2.model_definitions) –
- [**parameteriser**](#swift2.parameteriser) –
- [**play_record**](#swift2.play_record) –
- [**proto**](#swift2.proto) – Prototypes
- [**prototypes**](#swift2.prototypes) –
- [**simulation**](#swift2.simulation) –
- [**statistics**](#swift2.statistics) –
- [**system**](#swift2.system) –
- [**utils**](#swift2.utils) –
- [**vis**](#swift2.vis) –
- [**wrap**](#swift2.wrap) – CFFI based wrapper for the SWIFT2 native library

### swift2.classes

**Classes:**

- [**CandidateFactorySeed**](#swift2.classes.CandidateFactorySeed) –
- [**CompositeParameteriser**](#swift2.classes.CompositeParameteriser) – A parameteriser defined as the concatenation of several parameterisers
- [**ConstraintParameteriser**](#swift2.classes.ConstraintParameteriser) –
- [**EnsembleForecastSimulation**](#swift2.classes.EnsembleForecastSimulation) –
- [**EnsembleSimulation**](#swift2.classes.EnsembleSimulation) – A simulation designed to facilitate model runs over ensemble of inputs
- [**ErrisStagedCalibration**](#swift2.classes.ErrisStagedCalibration) –
- [**FilteringParameteriser**](#swift2.classes.FilteringParameteriser) –
- [**FunctionsParameteriser**](#swift2.classes.FunctionsParameteriser) –
- [**HypercubeParameteriser**](#swift2.classes.HypercubeParameteriser) –
- [**MaerrisStagedCalibration**](#swift2.classes.MaerrisStagedCalibration) –
- [**MemoryStates**](#swift2.classes.MemoryStates) –
- [**ObjectiveEvaluator**](#swift2.classes.ObjectiveEvaluator) – Objective Evaluator
- [**ObjectiveScores**](#swift2.classes.ObjectiveScores) –
- [**Optimiser**](#swift2.classes.Optimiser) –
- [**Parameteriser**](#swift2.classes.Parameteriser) – Wrapper around a native parameteriser.
- [**ScalingParameteriser**](#swift2.classes.ScalingParameteriser) –
- [**SceTerminationCondition**](#swift2.classes.SceTerminationCondition) –
- [**Simulation**](#swift2.classes.Simulation) – Wrapper around single dimension simulation objects
- [**SimulationMixin**](#swift2.classes.SimulationMixin) – A parent class for simulation objects. Most users are unlikely to explicitly use it.
- [**StateInitParameteriser**](#swift2.classes.StateInitParameteriser) –
- [**StateInitialiser**](#swift2.classes.StateInitialiser) –
- [**TransformParameteriser**](#swift2.classes.TransformParameteriser) –
- [**VectorObjectiveScores**](#swift2.classes.VectorObjectiveScores) –

**Functions:**

- [**wrap_cffi_native_handle**](#swift2.classes.wrap_cffi_native_handle) –

#### swift2.classes.CandidateFactorySeed

```python
CandidateFactorySeed(handle, release_native, type_id=None, prior_ref_count=0)
```

Bases: [DeletableCffiNativeHandle](#refcount.interop.DeletableCffiNativeHandle)

#### swift2.classes.CompositeParameteriser

```python
CompositeParameteriser(handle, release_native, type_id=None, prior_ref_count=0)
```

Bases: [HypercubeParameteriser](#swift2.classes.HypercubeParameteriser)

A parameteriser defined as the concatenation of several parameterisers

**Functions:**

- [**add_parameter_to_hypercube**](#swift2.classes.CompositeParameteriser.add_parameter_to_hypercube) – Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception
- [**add_to_hypercube**](#swift2.classes.CompositeParameteriser.add_to_hypercube) – Add entries to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception.
- [**append**](#swift2.classes.CompositeParameteriser.append) – Append a parameteriser to this composite parameteriser
- [**apply_sys_config**](#swift2.classes.CompositeParameteriser.apply_sys_config) – Apply a model configuration to a simulation
- [**as_dataframe**](#swift2.classes.CompositeParameteriser.as_dataframe) – Convert this hypercube parameteriser to a pandas data frame representation
- [**backtransform**](#swift2.classes.CompositeParameteriser.backtransform) – Get the parameteriser values in the untransformed space
- [**clone**](#swift2.classes.CompositeParameteriser.clone) –
- [**concatenate**](#swift2.classes.CompositeParameteriser.concatenate) – Concatenates some hypercubes to a single parameteriser
- [**create_parameter_sampler**](#swift2.classes.CompositeParameteriser.create_parameter_sampler) – Creates a sampler for this parameteriser
- [**empty_composite**](#swift2.classes.CompositeParameteriser.empty_composite) – Creates an empty parameteriser to be populated with other parameterisers
- [**filtered_parameters**](#swift2.classes.CompositeParameteriser.filtered_parameters) – Wrap a parameteriser in a filter that can hide some parameters
- [**from_dataframe**](#swift2.classes.CompositeParameteriser.from_dataframe) – Create a parameteriser
- [**hide_parameters**](#swift2.classes.CompositeParameteriser.hide_parameters) – Hide some parameters (from the outside e.g. optimisers) in a filter parameteriser
- [**make_state_init_parameteriser**](#swift2.classes.CompositeParameteriser.make_state_init_parameteriser) – Create a parameteriser used for model state initialisation
- [**num_free_parameters**](#swift2.classes.CompositeParameteriser.num_free_parameters) – Number of free parameters in this hypercube parameteriser
- [**score_for_objective**](#swift2.classes.CompositeParameteriser.score_for_objective) – Computes the value of an objective for this given set of parameters
- [**set_hypercube**](#swift2.classes.CompositeParameteriser.set_hypercube) – Set the properties of a hypercube parameteriser
- [**set_max_parameter_value**](#swift2.classes.CompositeParameteriser.set_max_parameter_value) – Sets the value(s) of the upper bound of one or more parameter(s)
- [**set_min_parameter_value**](#swift2.classes.CompositeParameteriser.set_min_parameter_value) – Sets the value(s) of the lower bound of one or more parameter(s)
- [**set_parameter_definition**](#swift2.classes.CompositeParameteriser.set_parameter_definition) – Sets the feasible range and value for a parameter
- [**set_parameter_value**](#swift2.classes.CompositeParameteriser.set_parameter_value) – Sets the value(s) of one or more parameter(s)
- [**show_parameters**](#swift2.classes.CompositeParameteriser.show_parameters) – Show some parameters (from the outside e.g. optimisers) in a filter parameteriser
- [**subcatchment_parameteriser**](#swift2.classes.CompositeParameteriser.subcatchment_parameteriser) – Create a parameteriser that gets applied to a subset of a whole catchment
- [**supports_thread_safe_cloning**](#swift2.classes.CompositeParameteriser.supports_thread_safe_cloning) – Is this parameteriser clonable as a deep copy, safe for multi-threading?
- [**wrap_transform**](#swift2.classes.CompositeParameteriser.wrap_transform) – Create a parameteriser for which parameter transformations can be defined.

##### swift2.classes.CompositeParameteriser.add_parameter_to_hypercube

```python
add_parameter_to_hypercube(name, value, min, max)
```

Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception

##### swift2.classes.CompositeParameteriser.add_to_hypercube

```python
add_to_hypercube(specs)
```

Add entries to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception.

**Parameters:**

- **specs** ([DataFrame](#pandas.DataFrame)) – An optional data frame description of the parameter set, with at least columns Name, Min, Max, Value.

##### swift2.classes.CompositeParameteriser.append

```python
append(p)
```

Append a parameteriser to this composite parameteriser

**Parameters:**

- **p** ([HypercubeParameteriser](#swift2.classes.HypercubeParameteriser)) – hypercube to append to this

##### swift2.classes.CompositeParameteriser.apply_sys_config

```python
apply_sys_config(simulation)
```

Apply a model configuration to a simulation

**Parameters:**

- **simulation** ([Simulation](#swift2.classes.Simulation)) – simulation

##### swift2.classes.CompositeParameteriser.as_dataframe

```python
as_dataframe()
```

Convert this hypercube parameteriser to a pandas data frame representation

**Returns:**

- [DataFrame](#pandas.DataFrame) – pd.DataFrame: pandas data frame

##### swift2.classes.CompositeParameteriser.backtransform

```python
backtransform()
```

Get the parameteriser values in the untransformed space

Get the parameteriser values in the untransformed space, i.e. remove any
transform added via \[`HypercubeParameteriser.wrap_transform`\][HypercubeParameteriser.wrap_transform].
This allows to transform back e.g. from a virtual parameter log_X
to the underlying model (or even virtual/meta) parameter X.

**Returns:**

- **HypercubeParameteriser** ([HypercubeParameteriser](#swift2.classes.HypercubeParameteriser)) – The parameters definitions without the transforms (if there are any)

**Examples:**

```pycon
>>> ref_area = 250
>>> time_span = 3600
>>> ptrans = sdh.define_gr4j_scaled_parameter(ref_area, time_span)
>>> ptrans
    Name     Value       Min       Max
0    log_x4  0.305422  0.000000  2.380211
1    log_x1  0.506690  0.000000  3.778151
2    log_x3  0.315425  0.000000  3.000000
3  asinh_x2  2.637752 -3.989327  3.989327
>>> ptrans.backtransform()
Name    Value   Min     Max
0   x2  6.95511 -27.0    27.0
1   x3  2.06740   1.0  1000.0
2   x4  2.02033   1.0   240.0
3   x1  3.21137   1.0  6000.0
>>>
```

##### swift2.classes.CompositeParameteriser.clone

```python
clone()
```

##### swift2.classes.CompositeParameteriser.concatenate

```python
concatenate(*args, strategy='')
```

Concatenates some hypercubes to a single parameteriser

**Parameters:**

- **strategy** ([str](#str)) – The strategy to contatenate. Defaults to "", equivalent to "composite", the only available. May have other options in the future.

Processing the swift2 API markdown format with `toolslm`

from toolslm.md_hier import *
hdgs = create_heading_dict(text)

subsection_header = hdgs['swift2']['swift2.classes']['swift2.classes.CompositeParameteriser']
subsection_header

{ 'swift2.classes.CompositeParameteriser.add_parameter_to_hypercube': {},
  'swift2.classes.CompositeParameteriser.add_to_hypercube': {},
  'swift2.classes.CompositeParameteriser.append': {},
  'swift2.classes.CompositeParameteriser.apply_sys_config': {},
  'swift2.classes.CompositeParameteriser.as_dataframe': {},
  'swift2.classes.CompositeParameteriser.backtransform': {},
  'swift2.classes.CompositeParameteriser.clone': {},
  'swift2.classes.CompositeParameteriser.concatenate': {},
  'swift2.classes.CompositeParameteriser.create_parameter_sampler': {},
  'swift2.classes.CompositeParameteriser.empty_composite': {},
  'swift2.classes.CompositeParameteriser.filtered_parameters': {},
  'swift2.classes.CompositeParameteriser.from_dataframe': {},
  'swift2.classes.CompositeParameteriser.hide_parameters': {},
  'swift2.classes.CompositeParameteriser.make_state_init_parameteriser': {},
  'swift2.classes.CompositeParameteriser.num_free_parameters': {},
  'swift2.classes.CompositeParameteriser.score_for_objective': {},
  'swift2.classes.CompositeParameteriser.set_hypercube': {},
  'swift2.classes.CompositeParameteriser.set_max_parameter_value': {},
  'swift2.classes.CompositeParameteriser.set_min_parameter_value': {},
  'swift2.classes.CompositeParameteriser.set_parameter_definition': {},
  'swift2.classes.CompositeParameteriser.set_parameter_value': {},
  'swift2.classes.CompositeParameteriser.show_parameters': {},
  'swift2.classes.CompositeParameteriser.subcatchment_parameteriser': {},
  'swift2.classes.CompositeParameteriser.supports_thread_safe_cloning': {},
  'swift2.classes.CompositeParameteriser.wrap_transform': {}}

AI Prompt

Massaging the hierarchical subsection to short names

The dictionary of dictionaries hdgs is such that the dictionary keys at every nested level have fully qualfied names, for instance hdgs['swift2']['swift2.classes']['swift2.classes.CompositeParameteriser']['swift2.classes.CompositeParameteriser.add_parameter_to_hypercube'] I want to process this dictionary hdgs and produce a new dictionary using short module and function names at every level, to that I have a new dictionary of dictionaries new_hdgs where I can address the nested content with new_hdgs['swift2']['classes']['CompositeParameteriser']['add_parameter_to_hypercube'] to access the final nested value, which is an empty dictionary {}. I suggest that in this case it is OK to assume that we can transform each sub-key by keeping the last dot-separated string, i.e. the sub-key 'swift2.classes.CompositeParameteriser.show_parameters' becomes 'show_parameters'. We do not need a more complicated algorithm that checks on the parent key of the sub-key to strip it from the sub-key. A possibly suitable strategy is to have a recursive python function to do this transformation, but I am open to your suggestions.

AI Response

I see what you’re trying to do! You want to transform your hierarchical dictionary to use shorter keys at each level, stripping away the fully qualified names.

Let’s create a recursive function to transform the dictionary. Here’s an approach we can take:

def shorten_keys(d, separator='.'):
    """
    Recursively transform dictionary keys to use only the last part after the separator.
    
    Args:
        d: Dictionary to transform
        separator: Character that separates parts of the key (default '.')
        
    Returns:
        New dictionary with shortened keys
    """
    result = {}
    
    for key, value in d.items():
        # Get the last part of the key after the separator
        short_key = key.split(separator)[-1]
        
        # If value is a dictionary, recursively process it
        if isinstance(value, dict):
            result[short_key] = shorten_keys(value, separator)
        else:
            result[short_key] = value
            
    return result

Would you like to try implementing this function? Or would you prefer me to show how to use it with your specific hdgs dictionary?

OK, this looks sound, let’s give this a try and look at the resulting nested dictionary

def shorten_keys(d, separator='.'):
    """
    Recursively transform dictionary keys to use only the last part after the separator.
    
    Args:
        d: Dictionary to transform
        separator: Character that separates parts of the key (default '.')
        
    Returns:
        New dictionary with shortened keys
    """
    result = {}
    
    for key, value in d.items():
        # Get the last part of the key after the separator
        short_key = key.split(separator)[-1]
        
        # If value is a dictionary, recursively process it
        if isinstance(value, dict):
            result[short_key] = shorten_keys(value, separator)
        else:
            result[short_key] = value
            
    return result

new_hdgs = shorten_keys(hdgs)

new_hdgs['swift2']['classes']['CompositeParameteriser']

{'add_parameter_to_hypercube': {},
 'add_to_hypercube': {},
 'append': {},
 'apply_sys_config': {},
 'as_dataframe': {},
 'backtransform': {},
 'clone': {},
 'concatenate': {},
 'create_parameter_sampler': {},
 'empty_composite': {},
 'filtered_parameters': {},
 'from_dataframe': {},
 'hide_parameters': {},
 'make_state_init_parameteriser': {},
 'num_free_parameters': {},
 'score_for_objective': {},
 'set_hypercube': {},
 'set_max_parameter_value': {},
 'set_min_parameter_value': {},
 'set_parameter_definition': {},
 'set_parameter_value': {},
 'show_parameters': {},
 'subcatchment_parameteriser': {},
 'supports_thread_safe_cloning': {},
 'wrap_transform': {}}

OK, this looks good. Now let us look at the dictionary that also includes the content, and flattens the markdown hierarchy

Massaging the dictionary of API content

hdict = markdown_to_dict(text)

first_subsections = list(hdict.keys())[:10]
first_subsections

['swift2',
 'swift2.swift2classes',
 'swift2.swift2classes.swift2classesCandidateFactorySeed',
 'swift2.swift2classes.swift2classesCompositeParameteriser',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriseraddparametertohypercube',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriseraddtohypercube',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriserappend',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriserapplysysconfig',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriserasdataframe',
 'swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriserbacktransform']

print(hdict['swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriseraddparametertohypercube'])

##### swift2.classes.CompositeParameteriser.add_parameter_to_hypercube

```python
add_parameter_to_hypercube(name, value, min, max)
```

Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception

AI Prompt

The implementation of the function markdown_to_dict is as follows:

import re
from fastcore.utils import dict2obj

def markdown_to_dict(markdown_content:str):
    def clean_heading(text): return re.sub(r'[^A-Za-z0-9 ]+', '', text).strip()

    lines = markdown_content.splitlines()
    headings = []
    in_code_block = False

    # Parse headings with their levels and line numbers
    for idx, line in enumerate(lines):
        # Toggle code block state when encountering fence
        if line.strip().startswith('```'): in_code_block = not in_code_block
        
        # Only detect headings when not in a code block
        if in_code_block: continue
        match = re.match(r'^(#{1,6})\s*(.*)', line)
        if match:
            level = len(match.group(1))
            text = match.group(2).strip()
            headings.append({'level': level, 'text': text, 'line': idx})

    # Assign content to each heading, including subheadings
    for i, h in enumerate(headings):
        start = h['line']  # Include the heading line itself
        # Find the end index: next heading of same or higher level
        for j in range(i + 1, len(headings)):
            if headings[j]['level'] <= h['level']:
                end = headings[j]['line']
                break
        else: end = len(lines)
        h['content'] = '\n'.join(lines[start:end]).strip()

    # Build the dictionary with hierarchical keys
    result,stack = {},[]
    first_level = headings[0]['level']
    for h in headings:
        stack = stack[:h['level'] - first_level] + [clean_heading(h['text'])]
        key = '.'.join(stack)
        result[key] = h['content']
    return dict2obj(result)

Now we observe that if we apply this function to the following string:

text_raw="""
## swift2

Python interface to the SWIFT2 streamflow simulation and forecasting libraries.

**Modules:**

- [**classes**](#swift2.classes) –

### swift2.classes

**Classes:**

- [**CandidateFactorySeed**](#swift2.classes.CandidateFactorySeed) –
- [**CompositeParameteriser**](#swift2.classes.CompositeParameteriser) – A parameteriser defined as the concatenation of several parameterisers

#### swift2.classes.CandidateFactorySeed

Bases: [DeletableCffiNativeHandle](#refcount.interop.DeletableCffiNativeHandle)

#### swift2.classes.CompositeParameteriser

Bases: [HypercubeParameteriser](#swift2.classes.HypercubeParameteriser)

A parameteriser defined as the concatenation of several parameterisers

**Functions:**

- [**add_parameter_to_hypercube**](#swift2.classes.CompositeParameteriser.add_parameter_to_hypercube) – Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception

##### swift2.classes.CompositeParameteriser.add_parameter_to_hypercube

Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception
"""

dict_sections = markdown_to_dict(text_raw)

Resulting from markdown_to_dict, dict_sections has a key ‘swift2.swift2classes.swift2classesCompositeParameteriser.swift2classesCompositeParameteriseraddparametertohypercube’, but what I want is dict_sections such that the key is ‘swift2.classes.CompositeParameteriser.add_parameter_to_hypercube’. Can you propose a modified markdown_to_dict such that I end up with the latter key, the fully qualified python name of the function swift2.classes.CompositeParameteriser.add_parameter_to_hypercube?

AI Response

I see the issue with the current markdown_to_dict function. It’s creating keys by cleaning the heading text (removing special characters) and then joining them, which results in those long concatenated keys without proper module structure.

Let’s modify the function to preserve the original heading structure instead of cleaning it. Here’s an approach:

import re
from fastcore.utils import dict2obj

def markdown_to_dict(markdown_content:str):
    lines = markdown_content.splitlines()
    headings = []
    in_code_block = False

    # Parse headings with their levels and line numbers
    for idx, line in enumerate(lines):
        # Toggle code block state when encountering fence
        if line.strip().startswith('```'): 
            in_code_block = not in_code_block
        
        # Only detect headings when not in a code block
        if in_code_block: 
            continue
            
        match = re.match(r'^(#{1,6})\s*(.*)', line)
        if match:
            level = len(match.group(1))
            text = match.group(2).strip()
            headings.append({'level': level, 'text': text, 'line': idx})

    # Assign content to each heading, including subheadings
    for i, h in enumerate(headings):
        start = h['line']  # Include the heading line itself
        # Find the end index: next heading of same or higher level
        for j in range(i + 1, len(headings)):
            if headings[j]['level'] <= h['level']:
                end = headings[j]['line']
                break
        else: 
            end = len(lines)
        h['content'] = '\n'.join(lines[start:end]).strip()

    # Build the dictionary with hierarchical keys
    result = {}
    for h in headings:
        # Use the original heading text as the key directly
        # This preserves the fully qualified names like "swift2.classes.CompositeParameteriser.add_parameter_to_hypercube"
        key = h['text']
        result[key] = h['content']
        
    return dict2obj(result)

Would you like me to explain how this differs from the original implementation, or would you prefer to test it first?

import re
from fastcore.utils import dict2obj

def my_markdown_to_dict(markdown_content:str):
    lines = markdown_content.splitlines()
    headings = []
    in_code_block = False

    # Parse headings with their levels and line numbers
    for idx, line in enumerate(lines):
        # Toggle code block state when encountering fence
        if line.strip().startswith('```'): 
            in_code_block = not in_code_block
        
        # Only detect headings when not in a code block
        if in_code_block: 
            continue
            
        match = re.match(r'^(#{1,6})\s*(.*)', line)
        if match:
            level = len(match.group(1))
            text = match.group(2).strip()
            headings.append({'level': level, 'text': text, 'line': idx})

    # Assign content to each heading, including subheadings
    for i, h in enumerate(headings):
        start = h['line']  # Include the heading line itself
        # Find the end index: next heading of same or higher level
        for j in range(i + 1, len(headings)):
            if headings[j]['level'] <= h['level']:
                end = headings[j]['line']
                break
        else: 
            end = len(lines)
        h['content'] = '\n'.join(lines[start:end]).strip()

    # Build the dictionary with hierarchical keys
    result = {}
    for h in headings:
        # Use the original heading text as the key directly
        # This preserves the fully qualified names like "swift2.classes.CompositeParameteriser.add_parameter_to_hypercube"
        key = h['text']
        result[key] = h['content']
        
    return dict2obj(result)

text_raw="""
## swift2

Python interface to the SWIFT2 streamflow simulation and forecasting libraries.

**Modules:**

- [**classes**](#swift2.classes) –

### swift2.classes

**Classes:**

- [**CandidateFactorySeed**](#swift2.classes.CandidateFactorySeed) –
- [**CompositeParameteriser**](#swift2.classes.CompositeParameteriser) – A parameteriser defined as the concatenation of several parameterisers

#### swift2.classes.CandidateFactorySeed

Bases: [DeletableCffiNativeHandle](#refcount.interop.DeletableCffiNativeHandle)

#### swift2.classes.CompositeParameteriser

Bases: [HypercubeParameteriser](#swift2.classes.HypercubeParameteriser)

A parameteriser defined as the concatenation of several parameterisers

**Functions:**

- [**add_parameter_to_hypercube**](#swift2.classes.CompositeParameteriser.add_parameter_to_hypercube) – Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception

##### swift2.classes.CompositeParameteriser.add_parameter_to_hypercube

Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception
"""

new_hdict = my_markdown_to_dict(text_raw)

print(list(new_hdict.keys()))

['swift2', 'swift2.classes', 'swift2.classes.CandidateFactorySeed', 'swift2.classes.CompositeParameteriser', 'swift2.classes.CompositeParameteriser.add_parameter_to_hypercube']

print(new_hdict['swift2.classes.CompositeParameteriser.add_parameter_to_hypercube'])

##### swift2.classes.CompositeParameteriser.add_parameter_to_hypercube

Add a parameter to a hypercube. Must be a type of object that is expandable, otherwise may raise an exception

OK, looking good, I think we may have a winner to process the API doc produced by griffe…