===================
django-data-browser
===================

**Interactive, user-friendly database explorer.**

.. image:: https://raw.githubusercontent.com/tolomea/django-data-browser/master/screenshot.png
    :alt: screenshot
    :align: center

.. contents::
    :depth: 1

Demo
----

There is a live demo site available here https://data-browser-demo.herokuapp.com/data-browser/.

Because it's hosted on Heroku free tier it might take a while to respond to the first page load.

The Django project is a small e-commerce site selling microservices.

Source: https://github.com/tolomea/data-browser-demo.

Admin: https://data-browser-demo.herokuapp.com/admin/.

Features
--------

* Zero config, if it's in the admin it's in the browser.
* Select fields (including calculated fields), aggregate, filter, sort and pivot.
* Automatically follow OneToOneFields and ForeignKeys.
* Respects per user admin permissions.
* Share views simply by sharing URLs.
* Save views and optionally make them available to services like Google sheets.
* Download views as CSV or JSON.

Supported Versions
------------------

The Data Browser is currently tested on:

* Django 3.2 - 5.0
* Python 3.8 - 3.12
* MySQL, PostgreSQL, SQLite

We highly recommend and only officially support the latest patch release of each Python and Django series.

Installation
------------

1. Run ``pip install django-data-browser``.
2. Add ``"data_browser"`` to installed_apps.
3. Add ``path("data-browser/", include("data_browser.urls"))`` to your urls.
4. Run ``python manage.py migrate``.
5. If you have queryset annotations in your admin or are interested in exposing calculated values see the `Calculated and Annotated fields`_ section.

Settings
--------

+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| Name                               | Default       | Docs Section                    | Function                                                                                |
+====================================+===============+=================================+====================================================================================================+
| ``DATA_BROWSER_ADMIN_FIELD_NAME``  | ``admin``     |                                 | Set the name for the "open in Django admin" field.                                                 |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_ALLOW_PUBLIC``      | ``False``     | `Security`_                     | Allow selected saved views to be accessed without admin login in limited circumstances.            |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_AUTH_USER_COMPAT``  | ``True``      | `Performance`_                  | When calling ``get_fieldsets`` on a ``UserAdmin`` always pass an instance of the associated model. |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_DEFAULT_ROW_LIMIT`` | ``1000``      |                                 | The default value for the row limit selector in the UI.                                            |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_DEV``               | ``False``     | CONTRIBUTING.rst                | Enable proxying frontend to JS dev server.                                                         |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_FE_DSN``            | ``None``      | `Sentry`_                       | The DSN the frontend sentry should report to, disabled by default.                                 |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_USING_DB``          | ``"default"`` | `Performance`_                  | Specify a database alias to use by default on all database queries.                                |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_ADMIN_OPTIONS``     | ``{}``        | `Specifying models and fields`_ | Set admin specific configuration options.                                                          |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_APPS_EXPANDED``     | ``True``      |                                 | Are the app sections of the homepage model list expanded by default.                               |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+
| ``DATA_BROWSER_ADMIN_SITE``        | ``None``      |  `Admin Site`_                  | Specify an ``admin.AdminSite`` to use (default is ``admin.site``).                                 |
+------------------------------------+---------------+---------------------------------+----------------------------------------------------------------------------------------------------+

Permissions
-----------

+------------------------------------+-------------+-------------------------------------------+
| ``data_browser.make_view_public``  | `Security`_ | Can make a saved view publicly available. |
+------------------------------------+-------------+-------------------------------------------+
| ``data_browser.share_view``        |             | Can share a saved view with other users.  |
+------------------------------------+-------------+-------------------------------------------+

Security
--------

Most of the Django views in the Data Browser can only be accessed by Django "staff members". These views support general querying of the database, checked against the admin permissions of the logged in user.

The only exception to this is "Public Saved Views" these are views which have been saved and marked as public. They can be accessed by anyone without needing a login but they can only be used to access a query that has been saved and made public and they have long random URL's.

You can use the admin permission ``data_browser | view | Can make a saved view publically available`` to restrict who can make views public. To be public the view must be marked as public and owned by someone who has the permission.

Additionally the entire public views system is gated by the Django settings value ``DATA_BROWSER_ALLOW_PUBLIC``.

Sentry
------

The frontend code has builtin Sentry support, it is **disabled by default**. To enable it set the Django settings value ``DATA_BROWSER_FE_DSN``, for example to set it to the Data Browser project Sentry use:

.. code-block:: python

    DATA_BROWSER_FE_DSN = "https://af64f22b81994a0e93b82a32add8cb2b@o390136.ingest.sentry.io/5231151"

Linking to the Data Browser
---------------------------

The home page URL of the Data Browser is given by ``reverse("data_browser:home")``.

Additionally if you are using ``data_browser.helpers.AdminMixin`` then in Admin list views the URL of the Data Browser page for the same model is available as the template context variable ``ddb_url``.

One convenient way of utilizing this is to create the file ``templates/admin/change_list_object_tools.html`` and populate it with:

.. code-block:: html

    {% extends "admin/change_list_object_tools.html" %}
    {% block object-tools-items %}
        {{ block.super }}
        {% if ddb_url %}

Data Browser

        {% endif %}
    {% endblock %}

This will place a "Data Browser" button on the list view of every admin that inherits from the mixin.
Note: to do this at the top level, the app you put the template in must be before contrib.admin in INSTALLED_APPS.

Specifying models and fields
----------------------------

By default the Data Browser has access to all models and fields that the current user can see anywhere in the Admin site.
However if necessary this can be tweaked using the following class level properties and functions on ``ModelAdmin``\s and ``Inline``\s.

+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
|   Name                              | Format                                    | Purpose                                                                                |
+=====================================+===========================================+=============================================================================================================+
| | ``ddb_ignore``                    | ``bool``                                  | Ignore this Admin / Inline entirely, will still show fields from other Inlines / Admins on the same model.  |
| | ``get_ddb_ignore(request)``       |                                           |                                                                                |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| | ``ddb_hide_fields``               | ``[field_name]``                          | Explicitly hide the specified fields.                                                                       |
| | ``get_ddb_hide_fields(request)``  |                                           |                                                                                |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| | ``ddb_extra_fields``              | ``[field_name]``                          | Add additional fields that are not mentioned in fields, fieldsets or list_display.                          |
| | ``get_ddb_extra_fields(request)`` |                                           |                                                                                |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| | ``ddb_json_fields``               | ``{field_name: {json_field_name: type}}`` | Expose fields within JSON data for access in the Data Browser. Type can be "string", "number" or "boolean". |
| | ``get_ddb_json_fields(request)``  |                                           |                                                                                |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| | ``ddb_default_filters``           | ``[(path, lookup, value)]``               | | Default filters to be added when opening this model.                                                      |
| | ``get_ddb_default_filters()``     |                                           | | E.G. to add ``client__name__equals=Test`` use ``[("client__name", "equals", "Test")]``.                   |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+
| | ``ddb_action_url``                | ``str``                                   | The url to post admin actions to, usually the changelist view. See `Admin Actions`_                         |
| | ``get_ddb_action_url(request)``   |                                           |                                                                                |
+-------------------------------------+-------------------------------------------+-------------------------------------------------------------------------------------------------------------+

These can also be set via the settings entry ``DATA_BROWSER_ADMIN_OPTIONS``, this is useful if you wish to change an option on a third party admin.
The format for this is ``{'my_package.MyAdmin': {'option': value}}``, for example to remove Django Q's Task and Fail admins entirely you would use:

.. code-block:: python

    DATA_BROWSER_ADMIN_OPTIONS = {
        'django_q.admin.TaskAdmin': {'ignore': True},
        'django_q.admin.FailAdmin': {'ignore': True},
    }

Finally, per the below sections, calculated fields and actions can be hidden by setting the ``ddb_hide`` attribute and annotated fields are always visible unless explicitly hidden.

Calculated and Annotated fields
-------------------------------

Calculated
^^^^^^^^^^

Calculated fields are fields on the ``ModelAdmin`` whose value comes from a function on the ``ModelAdmin`` or a function or property on the Model itself, as described at the bottom of the `Django admin list display docs <https://docs.djangoproject.com/en/stable/ref/contrib/admin/#django.contrib.admin.ModelAdmin.list_display>`_.

Being arbitrary Python code calculated fields are opaque to the Data Browser. It can fetch their values but can't sort or filter etc on them. For pivoting they are treated as equivalent to the pk on the same model.

Additionally calculated fields can be hidden from the Data Browser by setting the attribute ``ddb_hide`` to ``True``. The ``data_browser.helpers.attributes`` decorator can make this a little tidier.

.. code-block:: python

    @attributes(ddb_hide=True)
    def my_calculated_field(self, obj):
        return ...

Annotated
^^^^^^^^^

The Data Browser has additional support for annotated fields. Normally you would expose these as calculated fields. The module ``data_browser.helpers`` contains helpers which will make exposing annotated fields simpler, more performant and expose them to the Data Browser so it can do arbitrary manipulation with them.

Exposing an annotated field in this way requires two changes.

1. Mix ``data_browser.helpers.AdminMixin`` into your ModelAdmin.
2. Add a function decorated with ``data_browser.helpers.annotation`` that takes and updates a queryset.

.. code-block:: python

    from data_browser.helpers import AdminMixin, annotation

    @admin.register(MyModel)
    class MyAdmin(AdminMixin, ModelAdmin):
        fields = ["my_field"]

        @annotation
        def my_field(self, request, qs):
            return qs.annotate(my_field=Cast(..., output_field=IntegerField()))

WARNING: annotated aggregations will produce misleading results when further aggregated in the Data Browser.

It is important that the decorated annotation function name and the annotated queryset field name match.

Sometimes it is necessary for the top level of the annotation to have ``output_field`` set so the Data Browser can tell what type of data it will produce. When this is necessary you will get an error to that effect.

The helpers will automatically deal with the ``admin_order_field`` and ``boolean`` properties and ``readonly_fields``, reducing the boiler plate involved in using annotations in the admin.

Additionally the annotation will only be applied to the list view when it's mentioned in ``list_display`` this allows you to use annotations extensively on your detail views without hurting the performance of your list views.

And finally even if not mentioned in fields, fieldsets or list_display, the annotation will still be visible in the Data Browser unless it is explicitly mentioned in ``ddb_hide_fields``.

Performance
-----------

``get_queryset()``
^^^^^^^^^^^^^^^^^^

The Data Browser does it's fetching in two stages.

First it does a single DB query to get the majority of the data. To construct the queryset for this it will call ``get_queryset()`` on the ``ModelAdmin`` of the current Model. It uses ``.values()`` to fetch only the data it needs from the database and it will inline all referenced models to ensure it doesn't do multiple queries.

At this stage annotated fields on related models are attached with subquery annotations, the data browser will call ``get_queryset()`` on the relevant ``ModelAdmin``\s in order to generate these subquery annotations.

Secondly for any calculated fields it will then fetch the complete objects that are needed for those calculated fields. To construct the querysets for these it will call ``get_queryset()`` on their associated ``ModelAdmin``\s. These calls are aggregated so it will only make one per model.

As a simple example, if you did a query against the ``Book`` model for the fields:

* ``book.name``
* ``book.author.name``
* ``book.author.age``
* ``book.author.number_of_books``
* ``book.publisher.name``

Where the ``author.age`` is actually a property on the ``Author`` model, and ``author.number_of_books`` is an ``@annotation`` on the ``Author`` ``ModelAdmin``, then it would do something like the following two queries:

.. code-block:: python

    BookAdmin.get_queryset().annotate(
        author__number_of_books=Subquery(
            AuthorAdmin.get_queryset()
            .filter(pk=OuterRef("author__id"))
            .values("number_of_books")[:1]
        )
    ).values(
        "name",
        "author__name",
        "author__id",
        "author__number_of_books",
        "publisher__name",
    )
    AuthorAdmin.get_queryset().in_bulk(pks=...)

Where the ``pks`` passed to in_bulk in the second query came from ``author__id`` in the first.

You can view an approximation of the main queryset by changing the `.html` in the URL to `.qs`. In a similar manner `.sql`, `.explain` and `.analyze` are also available.

When the Data Browser calls the admin ``get_queryset()`` functions it will put some context in ``request.data_browser``. This allows you to test to see if the Data Browser is making the call as follows:

.. code-block:: python

    if hasattr(request, "data_browser"):
        # Data Browser specific customization

This is particularly useful if you want to route the Data Browser to a DB replica for a particular model (n.b. if you want to do this for all models see `QuerySet.using()`_ below.).

The context also includes a ``fields`` member that lists all the fields the Data Browser plans to access. You can use this to do conditional prefetching or annotating to support those fields like this:

.. code-block:: python

    if (
        not hasattr(request, "data_browser")
        or "my_field" in request.data_browser["fields"]
    ):
        # do prefetching and annotating associated with my_field

The AdminMixin described in the `Calculated and Annotated fields`_ section is doing this internally for ``@annotation`` fields.

QuerySet.using()
^^^^^^^^^^^^^^^^

The setting ``DATA_BROWSER_USING_DB`` can be used to direct Data Browser initiated database queries to a replica. Underneath the value of this is passed into ``QuerySet.using()``.

ModelAdmin.get_fieldsets()
^^^^^^^^^^^^^^^^^^^^^^^^^^

The Data Browser also calls ``get_fieldsets`` to find out what fields the current user can access.

As with ``get_queryset`` the Data Browser will set ``request.data_browser`` when calling ``get_fieldsets`` and you can test this to detect it and make Data Browser specific customizations.

The Django User Admin has code to change the fieldsets when adding a new user. To compensate for this, when calling ``get_fieldsets`` on a subclass of ``django.contrib.auth.admin.UserAdmin`` the Data Browser will pass a newly constructed instance of the relevant model. This behavior can be disabled by setting ``settings.DATA_BROWSER_AUTH_USER_COMPAT`` to ``False``.

Admin Actions
^^^^^^^^^^^^^

Django's Admin actions are exposed by right clicking on ID (or other appropriate pk field) column headers.

Due to the way these are implemented in Django there are some additional technical considerations.

The actions are posted to the Admin changelist URL. Once this post happens the Data Browser is no longer involved and so can't set ``request.data_browser`` like it normally would. Instead it will set the post argument ``data_browser``.

When the Data Browser triggers actions default Admin filtering is applied. If you have Admin filters that hide rows by default then actions triggered from the Data Browser will not be able to access those rows. To work around this you can specify ``get_ddb_action_url`` to override the URL the actions are posted to. By default it returns the changelist URL so you can append any arguments needed to set filters to not filter.

Style customization
-------------------

You can override the ``data_browser/index.html`` template per `Django’s template overriding docs <https://docs.djangoproject.com/en/stable/howto/overriding-templates/#extending-an-overridden-template>`__, and replace the ``extrahead`` block.
(Ensure ``"data_browser"`` is after your app in ``INSTALLED_APPS``.)

This will let you inject custom CSS and stylesheets.

However note that because of how the normal CSS is injected any custom CSS will be before the normal CSS so you will need to use more specific selectors or ``!important``.

Custom functions and aggregations
---------------------------------

It is possible to register additional functions and aggregations with the Data Browser, including custom ones.

Types
^^^^^

Functions and Aggregations are attached to the Data Browsers core types, these are in ``data_browser.types``.

Functions
^^^^^^^^^

*This functionality is provisional and not subject to normal backward compatible guarantees.*

The function registry is in ``data_browser.orm_functions.TYPE_FUNCTIONS`` this has type ``dict[BaseType, dict[str, data_browser.orm_functions.Func]]``. It is safe to insert new entries into this at runtime.

For example to add the MD5 function to string fields you could do the following:

.. code-block:: python

    from django.db.models.functions import MD5
    from data_browser.types import StringType
    from data_browser.orm_functions import Func, TYPE_FUNCTIONS

    TYPE_FUNCTIONS[StringType]["md5"] = Func(MD5, StringType)

Aggregates
^^^^^^^^^^

*This functionality is provisional and not subject to normal backward compatible guarantees.*

The aggregate registry is in ``data_browser.orm_aggregates.TYPE_AGGREGATES`` this has type ``dict[BaseType, dict[str, data_browser.orm_aggregates.Agg]]``. It is safe to insert new entries into this at runtime.

For example to add a count that does not apply distinct to number fields you could do the following:

.. code-block:: python

    from django.db.models import Count
    from data_browser.types import NumberType
    from data_browser.orm_aggregates import Agg, TYPE_AGGREGATES

    TYPE_AGGREGATES[NumberType]["full_count"] = Agg(
        lambda x: Count(x, distinct=False), NumberType
    )

Custom SQL example
^^^^^^^^^^^^^^^^^^

For a larger example imagine you wanted to use Postgres's ``percentile_cont`` functionality to add a ``p95`` aggregate to duration fields, perhaps for some kind of application performance monitoring usecase.

First we need to explain ``percentile_cont`` to Django.

.. code-block:: python

    from django.db.models import Aggregate

    class Percentile(Aggregate):
        arity = 1
        function = "percentile_cont"
        template = "%(function)s(%(percentile)s) WITHIN GROUP (ORDER BY %(expressions)s)"
        name = "Percentile"

        def __init__(self, percentile, expressions, **extra):
            super().__init__(expressions, percentile=percentile, **extra)

Then we need to tell the Data Browser we want ``p95`` on duration fields.

.. code-block:: python

    from data_browser.orm_aggregates import TYPE_AGGREGATES, Agg
    from data_browser.types import DurationType

    TYPE_AGGREGATES[DurationType]["p95"] = Agg(
        lambda x: Percentile(0.95, x), DurationType
    )

Admin Site
----------

You can create and use a custom ``admin.AdminSite`` (see https://docs.djangoproject.com/en/4.2/ref/contrib/admin/).

To do so, in your ``settings.py``, add:

.. code-block:: python

    from django.contrib import admin

    class BrowserAdminSite(admin.AdminSite):
        pass

    DATA_BROWSER_ADMIN_SITE = BrowserAdminSite(name='data_browser')

Then, in any ``admin.py``, register the models as usually but using ``DATA_BROWSER_ADMIN_SITE``.

For instance in ``myapp/admin.py``:

.. code-block:: python

    from django.contrib import admin
    from django.conf import settings
    from myapp.models import MyAdminModel, MyBrowsableModel

    # register in admin only
    admin.register(MyAdminModel)

    # register in data browser only
    settings.DATA_BROWSER_ADMIN_SITE.register(MyBrowsableModel)

Version numbers
---------------

The Data Browser uses the standard ``Major.Minor.Patch`` version numbering scheme.

Patch versions may include bug fixes and minor features.

Minor versions are for significant new features.

Major versions are for major features, significant changes to existing functionality and breaking changes.

Patch and Minor versions should never contain breaking changes and should always be backward compatible. A breaking change is a change that makes backward incompatible changes to one or more of the following:

* The query URL format.
* The json, csv etc data formats, this does not include the Data Browsers internal API's, only the data export formats.
* The format of the ``request.data_browser`` passed to ``get_fieldsets`` and ``get_queryset``.
* Existing saved views.
* The URL's of public saved views.

For alpha and beta releases absolutely anything may change / break.

Release History
---------------

+---------+------------+----------------------------------------------------------------------------------------------------------+
| Version | Date       | Summary                                                                                |
+=========+============+==========================================================================================================+
| 4.2.10  | 2024-04-08 | | Fix Z fighting issue with the field filter.                                                            |
|         |            | | Refactor JS for greater speed.                                                                         |
|         |            | | Fix a regex escaping issue in the field filter.                                                        |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.9   | 2024-04-02 | | Make field filter matching more permissive.                                                            |
|         |            | | Add a distinct (cyan) color for annotated fields.                                                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.8   | 2024-02-11 | | Improve field filter matching.                                                                         |
|         |            | | Add sticky headers to the field list.                                                                  |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.7   | 2024-02-04 | Add a text filter to the top of the field list.                                                          |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.6   | 2024-02-04 | | Support custom admin sites. Contributed by #aboutofpluto                                               |
|         |            | | Fix date expression in a filter on a datetime field not anchoring to midnight.                         |
|         |            | | Use verbose names for apps and models.                                                                 |
|         |            | | Add support for Python 3.12 and Django 5.0, drop support for Python 3.7.                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.5   | 2023-07-20 | | Fix parsing of date/datetime strings like ``mon-1``.                                                   |
|         |            | | Add support for DB query analyze via ``.analyze`` url, similar to the existing ``.explain``.           |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.4   | 2023-07-02 | | Provisional support for adding custom functions and aggregations.                                      |
|         |            | | Fix ``all`` aggregate on booleans and durations. (Postgres only)                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.3   | 2023-06-15 | Fix ASGI compatibility issue.                                                                            |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.2   | 2023-06-08 | Fix various issues around saved view validity.                                                           |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.2.1   | 2023-05-21 | | BREAKING: In JSON format move ``parsed`` and ``fitlerErrors`` onto the filters.                        |
|         |            | | Display invalid fields (previously they were ignored).                                                 |
|         |            | | Fix small bug when removing filters with errors.                                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.1.1   | 2023-05-07 | Fix bug with shared views when using multiple authentication backends.                                   |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.1.0   | 2023-05-01 | | Support sharing of saved views between users.                                                          |
|         |            | | Allow mixing of folders and views in the saved views section.                                          |
|         |            | | Fix bug where an invalid filter causes the query page to fail to render.                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.17  | 2023-04-25 | Fix bug with admin actions only working on the first column (introduced in 4.0.14).                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.16  | 2023-04-16 | | Group homepage model list into collapsible sections by app.                                            |
|         |            | | Allow grouping saved views into collapsible folders.                                                   |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.15  | 2023-04-11 | | Clear confirm prompts after a short delay.                                                             |
|         |            | | Remember the current saved view and allow updating it.                                                 |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.14  | 2023-04-05 | | Fix bug with admin actions not respecting ordering when the number of results exceeds the limit.       |
|         |            | | Add support for Django 4.2 and Python 3.11 drop support for Django 2.2, 3.0 and 3.1.                   |
|         |            | | Make "to many" support generally available.                                                            |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.13  | 2023-03-06 | | Add support for Django 4.1.                                                                            |
|         |            | | Work with CSRF_COOKIE_HTTPONLY which is enabled by cookiecutter-django.                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.12  | 2022-05-03 | Allow setting the per admin options via ``DATA_BROWSER_ADMIN_OPTIONS``.                                  |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.11  | 2022-04-12 | Fix bug when ``DATA_BROWSER_ADMIN_FIELD_NAME`` contains spaces.                                          |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.10  | 2022-04-10 | | Make results stable by always sorting all fields.                                                      |
|         |            | | Fix an issue with few to no results when pivoted and the headers exceed the result limit.              |
|         |            | | Add support for Postgres's array length function.                                                      |
|         |            | | Allow overriding the "open in Django admin" field name with ``DATA_BROWSER_ADMIN_FIELD_NAME``.         |
|         |            | | Add an AppConfig and declare the ``default_auto_field``.                                               |
|         |            | | Expose access to ``QuerySet.using()`` via a new ``DATA_BROWSER_USING_DB`` setting.                     |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.9   | 2022-01-04 | | Fix contains filter not working on files and URLs.                                                     |
|         |            | | Allow combining literal date time values with relative clauses.                                        |
|         |            | | Expose Postgres's ``ArrayAgg`` as an ``all`` aggregate.                                                |
|         |            | | Add mouse over text for the "to many" icon.                                                            |
|         |            | | Fix issue when ``related_name`` is different from ``related_query_name``                               |
|         |            | | CSS tweaks.                                                                                |
|         |            | | Change calculated field red to grey.                                                                   |
|         |            | | Add support for Django 4.0 and Python 3.10 drop support for Python 3.6.                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.8   | 2021-12-12 | | Fix formatting of ``F`` expressions when using ``.qs``.                                                |
|         |            | | Make the field list and filter list collapsible.                                                       |
|         |            | | CSS tweaks.                                                                                |
|         |            | | Add public view info to ``request.data_browser``.                                                      |
|         |            | | Fix crash when length filtering arrays of choice fields.                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.7   | 2021-08-16 | | Add support for django-hashid-field.                                                                   |
|         |            | | Fix a crash bug when aggregating fields with names starting with ``_``.                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.6   | 2021-08-10 | Fix spelling mistake.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.5   | 2021-08-09 | | Don't override right click context menu for HTML values (e.g. "Admin" columns).                        |
|         |            | | Fix "bad lookup" when excluding ``IsNull``/``NotNull`` values.                                         |
|         |            | | Fix pressing enter clearing all filters.                                                               |
|         |            | | Fix exceptions when using ``.qs``.                                                                     |
|         |            | | Fix view link on Saved View admin page not preserving ``limit``.                                       |
|         |            | | Improve placement of context menus.                                                                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.4   | 2021-07-04 | | Add ``.qs`` format support to see the main Django Queryset.                                            |
|         |            | | Support ``weeks`` in date and datetime filters.                                                        |
|         |            | | Fix bug filtering functions on annotations e.g. ``__my_annotation__is_null=IsNull``.                   |
|         |            | | Add admin actions to the admin column in addition to the id column.                                    |
|         |            | | Add exclude option to right click menus.                                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.3   | 2021-06-18 | Test on Django 3.2.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.2   | 2021-04-12 | URL, image and file fields filter like strings and render as strings in CSV and JSON.                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.1   | 2021-04-11 | | URLFields display as links.                                                                            |
|         |            | | Change pivot icon.                                                                                |
|         |            | | Automatically include the other side of OneToOne fields.                                               |
|         |            | | Disable custom context menus when right clicking inside a text selection.                              |
|         |            | | Fix rare issue with helpers.AdminMixin and MRO ordering of child classes.                              |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 4.0.0   | 2021-03-13 | | In the JSON output aggregate fields are now always in the body.                                        |
|         |            | | The CSV format has changed so aggregate fields are always to the right of other fields.                |
|         |            | | In the UI aggregate fields are now always to the right of other fields.                                |
|         |            | | Fields are colored by type, green: normal, blue: aggregates, red: can't sort or filter.                |
|         |            | | The right click drill down action now only adds filters where the row/column has multiple values.      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.3.0   | 2021-02-19 | | Drop support for Django 2.0 and 2.1                                                                    |
|         |            | | Rework Admin action integration.                                                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.5   | 2021-02-07 | | Date filter values formated as ``2020-1-2`` are now considered ISO ordered and no longer ambiguous.    |
|         |            | | Rework @annotation and AdminMixin so @annotation can be used on mixins.                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.4   | 2021-02-02 | | Fix ``equals`` and ``not equals`` not working for array fields.                                        |
|         |            | | Improve date and datetime filter errors.                                                               |
|         |            | | Improve and contrast display of null and empty string.                                                 |
|         |            | | Various fixes for models where the primary key is not ``id``.                                          |
|         |            | | Empty but non null file fields render as empty string instead of null.                                 |
|         |            | | Fix ``is null`` not working with the ``year`` function.                                                |
|         |            | | The field list is now sorted by display name (except for the primary key and admin link).              |
|         |            | | Fix ``not equals`` excluding nulls with functions and aggregates, e.g. ``year``, ``min`` etc.          |
|         |            | | Right click filter and drill down now correctly handle null values.                                    |
|         |            | | Prevent exception when a saved views name gets too long.                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.3   | 2021-01-11 | Fix issue when using a filter with a different type from the field, e.g. ``is null``.                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.2   | 2020-12-30 | | Fix ``id`` field missing from some models.                                                             |
|         |            | | Per Django, Django 2.0 & 2.1 are not supported on Py3.8 and 3.9.                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.1   | 2020-12-30 | Protect model admin class option values from accidental modification.                                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.2.0   | 2020-12-30 | | Support for invoking admin actions by right clicking on ``id`` column headers.                         |
|         |            | | Fix various filter issues.                                                                             |
|         |            | | Don't show ``id`` on models that don't have an ``id`` field.                                           |
|         |            | | Show "less than", "greater than" etc as "<", ">", etc.                                                 |
|         |            | | Mouse hover tooltip help for date and datetime filter values.                                          |
|         |            | | Filters with bad fields and lookups are reported as errors rather than being ignored.                  |
|         |            | | Bad filters on public saved View's now result in a 400 when loading the public URL.                    |
|         |            | | Fix issue filtering on aggregated annotations.                                                         |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.1.4   | 2020-12-19 | | Fix UUID's not being filterable.                                                                       |
|         |            | | Fix right click drill and filter trying to filter unfilterable fields.                                 |
|         |            | | Fix spurious ``0`` appearing below numeric ``0`` filter values.                                        |
|         |            | | Add an ``extrahead`` block to the template and documentation for overriding CSS.                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.1.3   | 2020-12-13 | | Relative time support in date and time filters.                                                        |
|         |            | | Show parsed dates and datetimes next to filters.                                                       |
|         |            | | Add view SQL link on front page. Contributed by #xinye1                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.1.2   | 2020-12-09 | | Remove length function from UUID's.                                                                    |
|         |            | | FK's with no admin are exposed as just the FK field.                                                   |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.1.1   | 2020-12-01 | Don't run the 3.0.0 data migration when there are no saved views.                                        |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.1.0   | 2020-11-29 | Add right click menu with filter and drill down options.                                                 |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.0.4   | 2020-11-28 | Ignore admins for things that are not Models.                                                            |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.0.3   | 2020-11-22 | Fix exception when filtering to out of bounds year values.                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.0.2   | 2020-11-18 | | Fix bug with aggregating around ``is null`` values on Django 3.1.                                      |
|         |            | | Fix ``is null`` returning None for missing fields in JsonFields.                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.0.1   | 2020-11-12 | | Add ``get_*`` functions for the ``ddb_*`` admin options.                                               |
|         |            | | Add length function to string fields.                                                                  |
|         |            | | Add support for DB query explain via ``.explain`` url.                                                 |
|         |            | | Prevent exception when getting SQL view of pure aggregates.                                            |
|         |            | | Fix incorrect handling of ISO dates whose day portion is less than 13.                                 |
|         |            | | Python 3.9 support.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 3.0.0   | 2020-11-09 | | The format of ddb_default_filters has changed.                                                         |
|         |            | | Path and prettyPath have been removed from fields and filters on JSON responses.                       |
|         |            | | Choice and ``is null`` fields use human readable values in filters.                                    |
|         |            | | Choice fields have a raw sub field for accessing the underlying values.                                |
|         |            | | Starts with, regex, etc have been removed form choice fields, equivalents are on raw.                  |
|         |            | | Verbose_names and short_descriptions are used for display in the web frontend and CSV.                 |
|         |            | | Equals and not equals for JSON and arrays.                                                             |
|         |            | | JSON field filter supports lists and objects.                                                          |
|         |            | | Array values are now JSON encoded across the board.                                                    |
|         |            | | Backfill saved views for above changes to filter formats.                                              |
|         |            | | Pickup calculated fields on inlines when there is no actual admin.                                     |
|         |            | | Fix bug where ID's and annotations on inlines were visible to users without perms.                     |
|         |            | | Support for aggregation and functions on annotated fields.                                             |
|         |            | | Annotations now respect ddb_hide.                                                                      |
|         |            | | Admin links to the Data Browser respect ddb_ignore.                                                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.21  | 2020-11-02 | Reject ambiguous date and datetime values in filters.                                                    |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.20  | 2020-10-22 | Fix bug with ``ArrayField`` on Django>=3.0                                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.19  | 2020-10-19 | Support for annotations on inlines.                                                                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.18  | 2020-10-18 | | Support for profiling CSV etc output. See CONTRIBUTING.rst                                             |
|         |            | | Performance improvements for large result sets.                                                        |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.17  | 2020-10-15 | | Performance improvements for large result sets.                                                        |
|         |            | | Fix error when choices field has an unexpected value.                                                  |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.16  | 2020-09-28 | | Fix being unable to reorder aggregates when there is no pivot.                                         |
|         |            | | Fix back button sometimes not remembering column reorderings.                                          |
|         |            | | Fix reordering columns while a long reload is in progress causes an error.                             |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.15  | 2020-09-27 | | Handle callables in ModelAdmin.list_display.                                                           |
|         |            | | Add ``data_browser.helpers.attributes``.                                                               |
|         |            | | Deprecated ``@ddb_hide`` in favor of ``@attributes(ddb_hide=True)``.                                   |
|         |            | | Render safestrings returned by calculated fields as HTML.                                              |
|         |            | | Respect the ``boolean`` attribute on calculated fields.                                                |
|         |            | | Aside from declared booleans, calculated fields now always format as strings.                          |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.14  | 2020-09-20 | | Saved view style tweaks.                                                                               |
|         |            | | Only reload on field delete when it might change the results.                                          |
|         |            | | Add UI controls for reordering fields.                                                                 |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.13  | 2020-09-13 | | Add ``.sql`` format to show raw SQL query.                                                             |
|         |            | | Min and max for date and datetime fields.                                                              |
|         |            | | Add ddb_default_filters.                                                                               |
|         |            | | Integrated cProfile support via ``.profile`` and ``.pstats``.                                          |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.12  | 2020-09-09 | | DurationField support.                                                                                |
|         |            | | Sort newly added date (etc) fields by default.                                                         |
|         |            | | Fix JSONField support when psycopg2 is not installed.                                                  |
|         |            | | Fix bug with number formatting and pivoted data.                                                       |
|         |            | | Fix error with multiple non adjacent filters on the same field.                                        |
|         |            | | Fix error with naive DateTimeFields.                                                                   |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.11  | 2020-08-31 | Minor enhancements and some small fixes.                                                                 |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.10  | 2020-08-31 | Minor enhancements.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.9   | 2020-08-25 | Small fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.8   | 2020-08-23 | Small fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.7   | 2020-08-22 | Small fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.6   | 2020-08-16 | Basic JSONField support.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.5   | 2020-08-01 | Bug fix.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.4   | 2020-08-01 | | Additional field support.                                                                              |
|         |            | | Minor features and bug fixes.                                                                          |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.3   | 2020-07-31 | File and Image field support                                                                             |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.2   | 2020-07-26 | Better support for choice fields.                                                                        |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.1   | 2020-07-25 | Performance tweaks.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.2.0   | 2020-07-21 | Sort and filter annotated fields.                                                                        |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.1.2   | 2020-07-11 | Minor bug fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.1.1   | 2020-07-06 | | Bug fixes.                                                                                |
|         |            | | The representation of empty pivot cells has changed in the JSON.                                       |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.1.0   | 2020-07-06 | | Bring views into the JS frontend.                                                                      |
|         |            | | Implement row limits on results.                                                                       |
|         |            | | All existing saved views will be limited to 1000 rows.                                                 |
|         |            | | Better loading and error status indication.                                                            |
|         |            | | Lock column headers.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.5   | 2020-06-20 | Bug fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.4   | 2020-06-18 | Fix Py3.6 support.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.3   | 2020-06-14 | Improve filtering on aggregates when pivoted.                                                            |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.2   | 2020-06-14 | Improve fonts and symbols.                                                                               |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.1   | 2020-06-14 | Improve sorting when pivoted.                                                                            |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 2.0.0   | 2020-06-14 | | Pivot tables.                                                                                |
|         |            | | All public view URL's have changed.                                                                    |
|         |            | | The JSON data format has changed.                                                                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.6   | 2020-06-08 | Bug fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.5   | 2020-06-08 | Bug fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.4   | 2020-06-03 | Calculated fields interact better with aggregation.                                                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.3   | 2020-06-02 | JS error handling tweaks.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.2   | 2020-06-01 | Minor fix.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.1   | 2020-05-31 | Improved date handling.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.2.0   | 2020-05-31 | Support for date functions "year", "month" etc and filtering based on "now".                             |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.6   | 2020-05-24 | Stronger sanitizing of URL strings.                                                                      |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.5   | 2020-05-23 | Fix bug aggregating time fields.                                                                         |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.4   | 2020-05-23 | Fix breaking bug with GenericInlineModelAdmin.                                                           |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.3   | 2020-05-23 | Cosmetic fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.2   | 2020-05-22 | Cosmetic fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.1   | 2020-05-20 | Cosmetic fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.1.0   | 2020-05-20 | Aggregate support.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.0.2   | 2020-05-17 | Py3.6 support.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.0.1   | 2020-05-17 | Small fixes.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+
| 1.0.0   | 2020-05-17 | Initial version.                                                                                |
+---------+------------+----------------------------------------------------------------------------------------------------------+


[![image](https://github.com/tefra/xsdata/raw/main/docs/logo.svg)](https://xsdata.readthedocs.io/)

# Naive XML Bindings for python

[![image](https://github.com/tefra/xsdata/workflows/tests/badge.svg)](https://github.com/tefra/xsdata/actions)
[![image](https://readthedocs.org/projects/xsdata/badge)](https://xsdata.readthedocs.io/)
[![image](https://codecov.io/gh/tefra/xsdata/branch/main/graph/badge.svg)](https://codecov.io/gh/tefra/xsdata)
[![image](https://www.codefactor.io/repository/github/tefra/xsdata/badge)](https://www.codefactor.io/repository/github/tefra/xsdata)
[![image](https://img.shields.io/pypi/pyversions/xsdata.svg)](https://pypi.org/pypi/xsdata/)
[![image](https://img.shields.io/pypi/v/xsdata.svg)](https://pypi.org/pypi/xsdata/)

---

xsData is a complete data binding library for python allowing developers to access and
use XML and JSON documents as simple objects rather than using DOM.

The code generator supports XML schemas, DTD, WSDL definitions, XML & JSON documents. It
produces simple dataclasses with type hints and simple binding metadata.

The included XML and JSON parser/serializer are highly optimized and adaptable, with
multiple handlers and configuration properties.

xsData is constantly tested against the
[W3C XML Schema 1.1 test suite](https://github.com/tefra/xsdata-w3c-tests).

## Getting started

```console
$ # Install all dependencies
$ pip install xsdata[cli,lxml,soap]
```

```console
$ # Generate models
$ xsdata tests/fixtures/primer/order.xsd --package tests.fixtures.primer
```

```python
>>> from tests.fixtures.primer import PurchaseOrder
>>> from xsdata.formats.dataclass.parsers import XmlParser
>>>
>>> parser = XmlParser()
>>> order = parser.parse("tests/fixtures/primer/sample.xml", PurchaseOrder)
>>> order.bill_to
Usaddress(name='Robert Smith', street='8 Oak Avenue', city='Old Town', state='PA', zip=Decimal('95819'), country='US')
```

Check the [documentation](https://xsdata.readthedocs.io) for more ✨✨✨

## Features

**Code Generator**

- XML Schemas 1.0 & 1.1
- WSDL 1.1 definitions with SOAP 1.1 bindings
- DTD external definitions
- Directly from XML and JSON Documents
- Extensive configuration to customize output
- Pluggable code writer for custom output formats

**Default Output**

- Pure python dataclasses with metadata
- Type hints with support for forward references and unions
- Enumerations and inner classes
- Support namespace qualified elements and attributes

**Data Binding**

- XML and JSON parser, serializer
- PyCode serializer
- Handlers and Writers based on lxml and native xml python
- Support wildcard elements and attributes
- Support xinclude statements and unknown properties
- Customize behaviour through config

## Changelog: 24.5 (2024-05-07)

**Features**

- Rewrite TreeSerializer, drop support for native python ElementTree
  ([#1032](https://github.com/tefra/xsdata/pull/1032))
- Validate fields fixed values ([#1013](https://github.com/tefra/xsdata/pull/1013))
- Detect optional fields in dict mapper
- Refactor typing annotations analyze process
- Generate ForwardRef() instead of Type[]

**Fixes**

- Allow soap client config subclassing
  ([#1010](https://github.com/tefra/xsdata/pull/1010))
- Avoid recursive error on nested group references
  ([#1016](https://github.com/tefra/xsdata/pull/1016))
- Add warning for same module designation
  ([#1018](https://github.com/tefra/xsdata/pull/1018))


![logo](https://raw.githubusercontent.com/DHI/mikeio/main/images/logo/PNG/MIKE-IO-Logo-Pos-RGB-nomargin.png)
# MIKE IO: input/output of MIKE files in Python
 ![Python version](https://img.shields.io/pypi/pyversions/mikeio.svg)
 [![Full test](https://github.com/DHI/mikeio/actions/workflows/full_test.yml/badge.svg)](https://github.com/DHI/mikeio/actions/workflows/full_test.yml)
[![PyPI version](https://badge.fury.io/py/mikeio.svg)](https://badge.fury.io/py/mikeio)
![OS](https://img.shields.io/badge/OS-Windows%20%7C%20Linux-blue)

Read, write and manipulate dfs0, dfs1, dfs2, dfs3, dfsu and mesh files.

MIKE IO facilitates common data processing workflows for [MIKE files in Python](https://www.mikepoweredbydhi.com/products/mike-for-developers#io).

[![MIKEIO. Read, write and analyze MIKE dfs files with Python on Vimeo](https://raw.githubusercontent.com/DHI/mikeio/main/images/youtube1.png)](https://player.vimeo.com/video/708275619)

<!--[![MIKEIO. New workflow and data structures in MIKE IO 1.0 on Vimeo](https://raw.githubusercontent.com/DHI/mikeio/main/images/youtube2.png)](https://player.vimeo.com/video/708276337)-->

## Requirements
* Windows or Linux operating system
* Python x64 3.8 - 3.12
* (Windows) [VC++ redistributables](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) (already installed if you have MIKE)

[More info about dependencies](http://docs.mikepoweredbydhi.com/nuget/)

## Installation

From PyPI:

`pip install mikeio`

Or development version:

`pip install https://github.com/DHI/mikeio/archive/main.zip`

:warning: **Don't use conda to install MIKE IO!**, the version on conda is outdated.

## Getting started

The material from the last Academy by DHI course is available here: [Getting started with Dfs files in Python using MIKE IO](https://dhi.github.io/getting-started-with-mikeio/intro.html)

## Where can I get help?
* Documentation - [https://dhi.github.io/mikeio/](https://dhi.github.io/mikeio/)
* General help, new ideas and feature requests - [GitHub Discussions](http://github.com/DHI/mikeio/discussions)
* Bugs - [GitHub Issues](http://github.com/DHI/mikeio/issues)

## Tested

MIKE IO is tested extensively.

See detailed test coverage report below:
```
---------- coverage: platform linux, python 3.11.1-final-0 -----------
Name                              Stmts   Miss  Cover
-----------------------------------------------------
mikeio/__init__.py                   38      2    95%
mikeio/base.py                       26      5    81%
mikeio/data_utils.py                111      5    95%
mikeio/dataarray.py                 992    152    85%
mikeio/dataset.py                   772    100    87%
mikeio/dfs0.py                      299     24    92%
mikeio/dfs1.py                       67      3    96%
mikeio/dfs2.py                      188      8    96%
mikeio/dfs3.py                      186     26    86%
mikeio/dfs.py                       288     36    88%
mikeio/dfsu/__init__.py               3      0   100%
mikeio/dfsu/dfsu.py                 592     56    91%
mikeio/dfsu/factory.py               41      2    95%
mikeio/dfsu/layered.py              180     20    89%
mikeio/dfsu/spectral.py             127      6    95%
mikeio/dfsutil.py                   129     15    88%
mikeio/eum.py                      1324     12    99%
mikeio/exceptions.py                 25      8    68%
mikeio/generic.py                   433     16    96%
mikeio/interpolation.py              60      5    92%
mikeio/pfs/__init__.py                4      0   100%
mikeio/pfs/pfsdocument.py           242     16    93%
mikeio/pfs/pfssection.py            220      9    96%
mikeio/spatial/FM_geometry.py      1118    102    91%
mikeio/spatial/FM_utils.py          293     30    90%
mikeio/spatial/__init__.py            0      0   100%
mikeio/spatial/crs.py                50      4    92%
mikeio/spatial/geometry.py           90     33    63%
mikeio/spatial/grid_geometry.py     558     42    92%
mikeio/spatial/utils.py              38      0   100%
mikeio/spectral.py                   90      5    94%
mikeio/track.py                     100     14    86%
mikeio/xyz.py                        12      0   100%
-----------------------------------------------------
TOTAL                              8696    756    91%

=========== 697 passed, 3 warnings in 46.01s ========
```

## Cloud enabled

It is possible to run MIKE IO in your favorite cloud notebook environment e.g. [Deepnote](https://deepnote.com/), [Google Colab](https://colab.research.google.com/), etc...

![DeepNote](https://raw.githubusercontent.com/DHI/mikeio/main/images/deepnote.png)

![Colab](https://raw.githubusercontent.com/DHI/mikeio/main/images/colab.png)


[![TerminusDB Python Client](https://assets.terminusdb.com/readmes/terminusdb-client-python/header.gif)][terminusdb-client-python-docs]

[terminusdb-client-python-docs]: https://terminusdb.com/docs/python

---

[![Discord](https://img.shields.io/discord/689805612053168129?label=Discord&logo=Discord&style=plastic)](https://discord.gg/yTJKAma)
[![Reddit](https://img.shields.io/reddit/subreddit-subscribers/TerminusDB?style=social)](https://www.reddit.com/r/TerminusDB/)
[![Twitter](https://img.shields.io/twitter/follow/terminusdb?color=skyblue&label=Follow%20on%20Twitter&logo=twitter&style=flat)](https://twitter.com/TerminusDB)

[![release version](https://img.shields.io/pypi/v/terminusdb-client.svg?logo=pypi)](https://pypi.python.org/pypi/terminusdb-client/)
[![downloads](https://img.shields.io/pypi/dm/terminusdb-client.svg?logo=pypi)](https://pypi.python.org/pypi/terminusdb-client/)

[![build status](https://img.shields.io/github/workflow/status/terminusdb/terminusdb-client-python/Python%20package?logo=github)](https://github.com/terminusdb/terminusdb-client-python/actions)
[![documentation](https://img.shields.io/github/deployments/terminusdb/terminusdb-client-python/github-pages?label=documentation&logo=github)](https://terminusdb.com/docs/python)
[![code coverage](https://codecov.io/gh/terminusdb/terminusdb-client-python/branch/main/graph/badge.svg?token=BclAUaOPnQ)](https://codecov.io/gh/terminusdb/terminusdb-client-python)
[![license](https://img.shields.io/github/license/terminusdb/terminusdb-client-python?color=pink&logo=apache)](https://github.com/terminusdb/terminusdb-client-python/blob/main/LICENSE)

> Python client for TerminusDB and TerminusCMS.

[**TerminusDB**][terminusdb] is an [open-source][terminusdb-repo] graph database
and document store. It allows you to link JSON documents in a powerful knowledge
graph all through a simple document API.

[terminusdb]: https://terminusdb.com/
[terminusdb-docs]: https://terminusdb.com/docs/
[terminusdb-repo]: https://github.com/terminusdb/terminusdb

[**TerminusCMS**](https://terminusdb.com/terminuscms/) is a hosted headless content management system. It is built upon TerminusDB and is a developer-focused data management platform for complex data and content infrastructure. [Sign up and clone a demo project to see how it works][dashboard].

[dashboard]: https://dashboard.terminusdb.com/

## Requirements

- [TerminusDB v10.0](https://github.com/terminusdb/terminusdb-server)
- [Python >=3.7](https://www.python.org/downloads)

## Release Notes and Previous Versions

TerminusDB Client v10.0 works with TerminusDB v10.0 and TerminusCMS. Please check the [Release Notes](RELEASE_NOTES.md) to find out what has changed.

## Installation
-  TerminusDB Client can be downloaded from PyPI using pip:
`python -m pip install terminusdb-client`

This only includes the core Python Client (Client) and WOQLQuery.

If you want to use woqlDataframe or the import and export CSV function in the Scaffolding CLI tool:

`python -m pip install terminusdb-client[dataframe]`

*if you are installing from `zsh` you have to quote the argument like this:*

`python -m pip install 'terminusdb-client[dataframe]'`

- Install from source:

`python -m pip install git+https://github.com/terminusdb/terminusdb-client-python.git`

## Demo

![Client Demo gif](https://github.com/terminusdb/terminusdb-web-assets/blob/master/images/terminusdb%20python%20v10%20client%20demo%201.gif)

## Usage

### Python client

#### Connect to a server

Connect to local host

```Python
from terminusdb_client import Client

client = Client("http://127.0.0.1:6363/")
client.connect()
```

Connect to TerminusCMS

*check the documentation for TerminusCMS about how to add the [API token](https://terminusdb.com/docs/how-to-connect-terminuscms) to the environment variable*

```Python
from terminusdb_client import Client

team="MyTeam"
client = Client(f"https://dashboard.terminusdb.com/{team}/")
client.connect(team="MyTeam", use_token=True)
```

#### Create a database

```Python
client.create_database("MyDatabase")
```

#### Create a schema

```Python
from terminusdb_client.schema import Schema, DocumentTemplate, RandomKey

my_schema = Schema()

class Pet(DocumentTemplate):
    _schema = my_schema
    name: str
    species: str
    age: int
    weight: float

my_schema.commit(client)
```

#### Create and insert documents

```Python
my_dog = Pet(name="Honda", species="Huskey", age=3, weight=21.1)
my_cat = Pet(name="Tiger", species="Bengal cat", age=5, weight=4.5)
client.insert_document([my_dog, my_cat])
```

#### Get back all documents

```Python
print(list(client.get_all_documents()))
```

```
[{'@id': 'Pet/b5edacf854e34fe79c228a91e2af45fb', '@type': 'Pet', 'age': 5, 'name': 'Tiger', 'species': 'Bengal cat', 'weight': 4.5}, {'@id': 'Pet/cdbe3f6d49394b38b952ae315309256d', '@type': 'Pet', 'age': 3, 'name': 'Honda', 'species': 'Huskey', 'weight': 21.1}]
```

#### Get a specific document

```Python
print(list(client.query_document({"@type":"Pet", "age":5})))
```

```
[{'@id': 'Pet/145eb73966d14a1394f7cd5576d7d0b8', '@type': 'Pet', 'age': 5, 'name': 'Tiger', 'species': 'Bengal cat', 'weight': 4.5}]
```

#### Delete a database

```Python
client.delete_database("MyDatabase")
```

### Scaffolding CLI tool

![Scaffolding Demo gif](https://github.com/terminusdb/terminusdb-web-assets/blob/master/images/terminusdb%20python%20v10%20scaffolding%20demo%202.gif)

Start a project in the directory

```bash
$ tdbpy startproject
Please enter a project name (this will also be the database name): mydb
Please enter an endpoint location (press enter to use localhost default) [http://127.0.0.1:6363/]:
config.json and schema.py created, please customize them to start your project.
```

Import a CSV named `grades.csv`

``` bash
$ tdbpy importcsv grades.csv --na=error
0it [00:00, ?it/s]
Schema object Grades created with grades.csv inserted into database.
schema.py is updated with mydb schema.
1it [00:00,  1.00it/s]
Records in grades.csv inserted as type Grades into database with Lexical ids.
```

Get documents with query

```bash
$ tdbpy alldocs --type Grades -q grade="B-"
[{'@id': 'Grades/Android_Electric_087-65-4321_42.0_23.0_36.0_45.0_47.0_B-', '@type': 'Grades', 'final': 47.0, 'first_name': 'Electric', 'grade': 'B-', 'last_name': 'Android', 'ssn': '087-65-4321', 'test1': 42.0, 'test2': 23.0, 'test3': 36.0, 'test4': 45.0}, {'@id': 'Grades/Elephant_Ima_456-71-9012_45.0_1.0_78.0_88.0_77.0_B-', '@type': 'Grades', 'final': 77.0, 'first_name': 'Ima', 'grade': 'B-', 'last_name': 'Elephant', 'ssn': '456-71-9012', 'test1': 45.0, 'test2': 1.0, 'test3': 78.0, 'test4': 88.0}, {'@id': 'Grades/Franklin_Benny_234-56-2890_50.0_1.0_90.0_80.0_90.0_B-', '@type': 'Grades', 'final': 90.0, 'first_name': 'Benny', 'grade': 'B-', 'last_name': 'Franklin', 'ssn': '234-56-2890', 'test1': 50.0, 'test2': 1.0, 'test3': 90.0, 'test4': 80.0}]
```

Delete the database

```bash
$ tdbpy deletedb
Do you want to delete 'mydb'? WARNING: This operation is non-reversible. [y/N]: y
mydb deleted.
```

### Please check the [full Documentation](https://terminusdb.com/docs/python) for more information.

## Guides & Tutorials

Visit our documentation for a range of short how-to guides, [how-to use the Python Client](https://terminusdb.com/docs/use-the-python-client) and [how to use the collaboration features with the Python Client](https://terminusdb.com/docs/collaboration-with-python-client). Alternatively, undertake the  [Getting Started with the Python Client Tutorial Series.](https://github.com/terminusdb/terminusdb-tutorials/blob/main/getting_started/python-client/README.md).

## Testing

1. Clone this repository
`git clone https://github.com/terminusdb/terminusdb-client-python.git`

2. Install all development dependencies using poetry
```sh
$ make init
```

3. (a) To run test files only
```sh
$ poetry run pytest terminusdb_client/tests
```

3. (b) To run full test
```sh
$ poetry run tox -e test
```

Or you can run tests in [poetry shell](https://python-poetry.org/docs/basic-usage/#activating-the-virtual-environment) then no need to use `poetry run`

## Documentation

Visit our [TerminusDB Documentation][terminusdb-docs] for the full explanation of using TerminusDB.

Documentation specifically on the latest version of the Python Client can be found [here][terminusdb-client-python-docs].

### Generating Documentation Locally using Sphinx

1. Clone this repository
`git clone https://github.com/terminusdb/terminusdb-client-python.git`

2. Install all development dependencies
```sh
$ make init
```

3. Change the directory to docs
```sh
$ cd docs/
```

4. Build with Sphinx
```sh
$ make html
```

The output files are under `docs/build/html`, open `index.html` in your browser to inspect.

## Report Issues

If you encounter any issues, please [report them](https://github.com/terminusdb/terminusdb-client-python/issues) with your os and environment setup, the version that you are using and a simple reproducible case.

If you have other questions, you can ask in our community [community Subreddit](https://www.reddit.com/r/TerminusDB/) or [Discord server](https://discord.gg/Gvdqw97).

## Community

Come visit us on our [discord server](https://discord.gg/yTJKAma)
or our [community Subreddit](https://www.reddit.com/r/TerminusDB/). We are also on [Twitter](https://twitter.com/TerminusDB)
<img align="right" src="https://assets.terminusdb.com/images/TerminusDB%20color%20mascot.png" width="256px"/>

## Contribute

It will be nice, if you open an issue first so that we can know what is going on, then, fork this repo and push in your ideas. Do not forget to add some test(s) of what value you adding.

Please check [CONTRIBUTING.md](CONTRIBUTING.md) for more information.

## Licence

Apache License (Version 2.0)

Copyright (c) 2019


# Python: WLED API Client

[![GitHub Release][releases-shield]][releases]
[![Python Versions][python-versions-shield]][pypi]
![Project Stage][project-stage-shield]
![Project Maintenance][maintenance-shield]
[![License][license-shield]](LICENSE.md)

[![Build Status][build-shield]][build]
[![Code Coverage][codecov-shield]][codecov]
[![Quality Gate Status][sonarcloud-shield]][sonarcloud]
[![Open in Dev Containers][devcontainer-shield]][devcontainer]

[![Sponsor Frenck via GitHub Sponsors][github-sponsors-shield]][github-sponsors]

[![Support Frenck on Patreon][patreon-shield]][patreon]

Asynchronous Python client for WLED.

## About

This package allows you to control and monitor an WLED device
programmatically. It is mainly created to allow third-party programs to automate
the behavior of WLED.

## Installation

```bash
pip install wled
```

## Usage

```python
import asyncio

from wled import WLED

async def main() -> None:
    """Show example on controlling your WLED device."""
    async with WLED("wled-frenck.local") as led:
        device = await led.update()
        print(device.info.version)

        # Turn strip on, full brightness
        await led.master(on=True, brightness=255)

if __name__ == "__main__":
    asyncio.run(main())
```

## Changelog & Releases

This repository keeps a change log using [GitHub's releases][releases]
functionality.

Releases are based on [Semantic Versioning][semver], and use the format
of `MAJOR.MINOR.PATCH`. In a nutshell, the version will be incremented
based on the following:

- `MAJOR`: Incompatible or major changes.
- `MINOR`: Backwards-compatible new features and enhancements.
- `PATCH`: Backwards-compatible bugfixes and package updates.

## Contributing

This is an active open-source project. We are always open to people who want to
use the code or contribute to it.

We've set up a separate document for our
[contribution guidelines](CONTRIBUTING.md).

Thank you for being involved! :heart_eyes:

## Setting up development environment

The easiest way to start, is by opening a CodeSpace here on GitHub, or by using
the [Dev Container][devcontainer] feature of Visual Studio Code.

[![Open in Dev Containers][devcontainer-shield]][devcontainer]

This Python project is fully managed using the [Poetry][poetry] dependency
manager. But also relies on the use of NodeJS for certain checks during
development.

You need at least:

- Python 3.11+
- [Poetry][poetry-install]
- NodeJS 20+ (including NPM)

To install all packages, including all development requirements:

```bash
npm install
poetry install --extras cli
```

As this repository uses the [pre-commit][pre-commit] framework, all changes
are linted and tested with each commit. You can run all checks and tests
manually, using the following command:

```bash
poetry run pre-commit run --all-files
```

To run just the Python tests:

```bash
poetry run pytest
```

## Authors & contributors

The original setup of this repository is by [Franck Nijhof][frenck].

For a full list of all authors and contributors,
check [the contributor's page][contributors].

## License

MIT License

Copyright (c) 2019-2024 Franck Nijhof

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

[build-shield]: https://github.com/frenck/python-wled/actions/workflows/tests.yaml/badge.svg
[build]: https://github.com/frenck/python-wled/actions/workflows/tests.yaml
[codecov-shield]: https://codecov.io/gh/frenck/python-wled/branch/main/graph/badge.svg
[codecov]: https://codecov.io/gh/frenck/python-wled
[contributors]: https://github.com/frenck/python-wled/graphs/contributors
[devcontainer-shield]: https://img.shields.io/static/v1?label=Dev%20Containers&message=Open&color=blue&logo=visualstudiocode
[devcontainer]: https://vscode.dev/redirect?url=vscode://ms-vscode-remote.remote-containers/cloneInVolume?url=https://github.com/frenck/python-wled
[frenck]: https://github.com/frenck
[github-sponsors-shield]: https://frenck.dev/wp-content/uploads/2019/12/github_sponsor.png
[github-sponsors]: https://github.com/sponsors/frenck
[license-shield]: https://img.shields.io/github/license/frenck/python-wled.svg
[maintenance-shield]: https://img.shields.io/maintenance/yes/2024.svg
[patreon-shield]: https://frenck.dev/wp-content/uploads/2019/12/patreon.png
[patreon]: https://www.patreon.com/frenck
[poetry-install]: https://python-poetry.org/docs/#installation
[poetry]: https://python-poetry.org
[pre-commit]: https://pre-commit.com/
[project-stage-shield]: https://img.shields.io/badge/project%20stage-experimental-yellow.svg
[pypi]: https://pypi.org/project/wled/
[python-versions-shield]: https://img.shields.io/pypi/pyversions/wled
[releases-shield]: https://img.shields.io/github/release/frenck/python-wled.svg
[releases]: https://github.com/frenck/python-wled/releases
[semver]: http://semver.org/spec/v2.0.0.html
[sonarcloud-shield]: https://sonarcloud.io/api/project_badges/measure?project=frenck_python-wled&metric=alert_status
[sonarcloud]: https://sonarcloud.io/summary/new_code?id=frenck_python-wled