# TODO ugh. my blog generator dumps links as file: ....
# so used smeth like :s/file:\(.*\)\.org/https:\/\/beepb00p.xyz\/\1.html/gc -- steal leaves ::# links etc. ugh
#+summary: My life in a Python package
#+created: [2019-11-14 Thu]
#+filetags: :infra:pkm:quantifiedself:hpi:
#+upid: mypkg

#+macro: map      @@html:@@$1@@html:@@

If you're in a hurry, feel free to jump straight to the [[#usecases][demos]].

- see [[https://github.com/karlicoss/HPI/tree/master/doc/SETUP.org][SETUP]] for the *installation/configuration guide*
- see [[https://github.com/karlicoss/HPI/tree/master/doc/DEVELOPMENT.org][DEVELOPMENT]] for the *development guide*
- see [[https://github.com/karlicoss/HPI/tree/master/doc/DESIGN.org][DESIGN]] for the *design goals*
- see [[https://github.com/karlicoss/HPI/tree/master/doc/MODULES.org][MODULES]] for *module-specific setup*
- see [[https://github.com/karlicoss/HPI/tree/master/doc/MODULE_DESIGN.org][MODULE_DESIGN]] for some thoughts on structuring modules, and possibly *extending HPI*
- see [[https://beepb00p.xyz/exobrain/projects/hpi.html][exobrain/HPI]] for some of my raw thoughts and todos on the project

*TLDR*: I'm using [[https://github.com/karlicoss/HPI][HPI]] (Human Programming Interface) package as a means of unifying, accessing and interacting with all of my personal data.

HPI is a Python package (named ~my~), a collection of modules for:

- social networks: posts, comments, favorites
- reading: e-books and pdfs
- annotations: highlights and comments
- todos and notes
- health data: sleep, exercise, weight, heart rate, and other body metrics
- location
- photos & videos
- browser history
- instant messaging

The package hides the [[https://beepb00p.xyz/sad-infra.html#exports_are_hard][gory details]] of locating data, parsing, error handling and caching.
You simply 'import' your data and get to work with familiar Python types and data structures.

- Here's a short example to give you an idea: "which subreddits I find the most interesting?"

  #+begin_src python
    import my.reddit.all
    from collections import Counter
    return Counter(s.subreddit for s in my.reddit.all.saved()).most_common(4)
  #+end_src

  | orgmode        | 62 |
  | emacs          | 60 |
  | selfhosted     | 51 |
  | QuantifiedSelf | 46 |

I consider my digital trace an important part of my identity. ([[https://beepb00p.xyz/tags.html#extendedmind][#extendedmind]])
Usually the data is siloed, accessing it is inconvenient and borderline frustrating. This feels very wrong.

In contrast, once the data is available as Python objects, I can easily plug it into existing tools, libraries and frameworks.
It makes building new tools considerably easier and opens up new ways of interacting with the data.

I tried different things over the years and I think I'm getting to the point where other people can also benefit from my code by 'just' plugging in their data,
and that's why I'm sharing this.

Imagine if all your life was reflected digitally and available at your fingertips.
This library is my attempt to achieve this vision.

#+toc: headlines 2

:results:
*Table of contents:*
- Why?
- How does a Python package help?
  - Why don't you just put everything in a massive database?
- What's inside?
- How do you use it?
- Ad-hoc and interactive
  - What were my music listening stats for 2018?
  - What are the most interesting Slate Star Codex posts I've read?
  - Accessing exercise data
  - Book reading progress
  - Messenger stats
  - Which month in 2020 did I make the most git commits in?
  - Querying Roam Research database
- How does it get input data?
- Q & A
  - Why Python?
  - Can anyone use it?
  - How easy is it to use?
  - What about privacy?
  - But /should/ I use it?
  - Would it suit /me/?
  - What it isn't?
- HPI Repositories
- Related links
- --
:END:

* Why?
:PROPERTIES:
:CUSTOM_ID: motivation
:END:

The main reason that led me to develop this is the dissatisfaction of the current situation:

- Our personal data is siloed and trapped across cloud services and various devices

  Even when it's possible to access it via the API, it's hardly useful, unless you're an experienced programmer, willing to invest your time and infrastructure.

- We have insane amounts of data scattered across the cloud, yet we're left at the mercy of those who collect it to provide something useful based on it

  Integrations of data across silo boundaries are almost non-existent. There is so much potential and it's all wasted.

- I'm not willing to wait till some vaporware project reinvents the whole computing model from scratch

  As a programmer, I am in capacity to do something *right now*, even though it's not necessarily perfect and consistent.

I've written a lot about it [[https://beepb00p.xyz/sad-infra.html#why][here]], so allow me to simply quote:

:results:
#+begin_quote
- search and information access
  - Why can't I search over all of my personal chat history with a friend, whether it's ICQ logs from 2005 or Whatsapp logs from 2019?
  - Why can't I have incremental search over my tweets? Or browser bookmarks? Or over everything I've ever typed/read on the Internet?
  - Why can't I search across my watched youtube videos, even though most of them have subtitles hence allowing for full text search?
  - Why can't I see the places my friends recommended me on Google maps (or any other maps app)?
- productivity
  - Why can't my Google Home add shopping list items to Google Keep? Let alone other todo-list apps.
  - Why can't I create a task in my todo list or calendar from a conversation on Facebook Messenger/Whatsapp/VK.com/Telegram?
- journaling and history
  - Why do I have to lose all my browser history if I decide to switch browsers?
  - Why can't I see all the places I traveled to on a single map and photos alongside?
  - Why can't I see what my heart rate (i.e. excitement) and speed were side by side with the video I recorded on GoPro while skiing?
  - Why can't I easily transfer all my books and metadata if I decide to switch from Kindle to PocketBook or vice versa?
- consuming digital content
  - Why can't I see stuff I highlighted on Instapaper as an overlay on top of web page?
  - Why can't I have single 'read it later' list, unifying all things saved on Reddit/Hackernews/Pocket?
  - Why can't I use my todo app instead of 'Watch later' playlist on youtube?
  - Why can't I 'follow' some user on Hackernews?
  - Why can't I see if I've run across a Youtube video because my friend sent me a link months ago?
  - Why can't I have uniform music listening stats based on my Last.fm/iTunes/Bandcamp/Spotify/Youtube?
  - Why am I forced to use Spotify's music recommendation algorithm and don't have an option to try something else?
  - Why can't I easily see what were the books/music/art recommended by my friends or some specific Twitter/Reddit/Hackernews users?
  - Why my otherwise perfect hackernews [[https://play.google.com/store/apps/details?id=io.github.hidroh.materialistic][app for Android]] doesn't share saved posts/comments with the website?
- health and body maintenance
  - Why can't I tell if I was more sedentary than usual during the past week and whether I need to compensate by doing a bit more exercise?
  - Why can't I see what's the impact of aerobic exercise on my resting HR?
  - Why can't I have a dashboard for all of my health: food, exercise and sleep to see baselines and trends?
  - Why can't I see the impact of temperature or CO2 concentration in room on my sleep?
  - Why can't I see how holidays (as in, not going to work) impact my stress levels?
  - Why can't I take my Headspace app data and see how/if meditation impacts my sleep?
  - Why can't I run a short snippet of code and check some random health advice on the Internet against *my* health data.
- personal finance
  - Why am I forced to manually copy transactions from different banking apps into a spreadsheet?
  - Why can't I easily match my Amazon/Ebay orders with my bank transactions?
- why I can't do anything when I'm offline or have a wonky connection?
- tools for thinking and learning
  - Why when something like [[https://en.wikipedia.org/wiki/Method_of_loci]['mind palace']] is *literally possible* with VR technology, we don't see any in use?
  - Why can't I easily convert select Instapaper highlights or new foreign words I encountered on my Kindle into Anki flashcards?
- mediocre interfaces
  - Why do I have to suffer from poor management and design decisions in UI changes, even if the interface is not the main reason I'm using the product?
  - Why can't I leave priorities and notes on my saved Reddit/Hackernews items?
  - Why can't I leave private notes on Deliveroo restaurants/dishes, so I'd remember what to order/not to order next time?
  - Why do people have to suffer from Google Inbox shutdown?
- communication and collaboration
  - Why can't I easily share my web or book highlights with a friend? Or just make highlights in select books public?
  - Why can't I easily find out other person's expertise without interrogating them, just by looking what they read instead?
- backups
  - Why do I have to think about it and actively invest time and effort?
#+end_quote
:END:

- I'm tired of having to use multiple different messengers and social networks
- I'm tired of shitty bloated interfaces

  Why do we have to be at mercy of their developers, designers and product managers? If we had our data at hand, we could fine-tune interfaces for our needs.

- I'm tired of mediocre search experience

  Text search is something computers do *exceptionally* well.
  Yet, often it's not available offline, it's not incremental, everyone reinvents their own query language, and so on.

- I'm frustrated by poor information exploring and processing experience

  While for many people, services like Reddit or Twitter are simply time killers (and I don't judge), some want to use them efficiently, as a source of information/research.
  Modern bookmarking experience makes it far from perfect.

You can dismiss this as a list of first-world problems, and you would be right, they are.
But the major reason I want to solve these problems is to be better at learning and working with knowledge,
so I could be better at solving the real problems.

* How does a Python package help?
:PROPERTIES:
:CUSTOM_ID: package
:END:

When I started solving some of these problems for myself, I've noticed a common pattern: the [[https://beepb00p.xyz/sad-infra.html#exports_are_hard][hardest bit]] is actually getting your data in the first place.
It's inherently error-prone and frustrating.

But once you have the data in a convenient representation, working with it is pleasant -- you get to *explore and build instead of fighting with yet another stupid REST API*.

This package knows how to find data on your filesystem, deserialize it and normalize it to a convenient representation.
You have the full power of the programming language to transform the data and do whatever comes to your mind.

** Why don't you just put everything in a massive database?
:PROPERTIES:
:CUSTOM_ID: database
:END:
Glad you've asked! I wrote a whole [[https://beepb00p.xyz/unnecessary-db.html][post]] about it.

In short: while databases are efficient and easy to read from, often they aren't flexible enough to fit your data.
You're probably going to end up writing code anyway.

While working with your data, you'll inevitably notice common patterns and code repetition, which you'll probably want to extract somewhere.
That's where a Python package comes in.

* What's inside?
:PROPERTIES:
:CUSTOM_ID: modules
:END:

Here's the (incomplete) list of the modules:

:results:
| [[https://github.com/karlicoss/HPI/tree/master/my/bluemaestro.py][=my.bluemaestro=]]                                 | [[https://bluemaestro.com/products/product-details/bluetooth-environmental-monitor-and-logger][Bluemaestro]] temperature/humidity/pressure monitor |
| [[https://github.com/karlicoss/HPI/tree/master/my/body/blood.py][=my.body.blood=]]                                   | Blood tracking (manual org-mode entries)                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/body/exercise/all.py][=my.body.exercise.all=]]                     | Combined exercise data                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/body/exercise/cardio.py][=my.body.exercise.cardio=]]               | Cardio data, filtered from various data sources                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/body/exercise/cross_trainer.py][=my.body.exercise.cross_trainer=]] | My cross trainer exercise data, arbitrated from different sources (mainly, Endomondo and manual text notes)                                        |
| [[https://github.com/karlicoss/HPI/tree/master/my/body/weight.py][=my.body.weight=]]                                 | Weight data (manually logged)                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/calendar/holidays.py][=my.calendar.holidays=]]                     | Holidays and days off work                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/coding/commits.py][=my.coding.commits=]]                           | Git commits data for repositories on your filesystem                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/demo.py][=my.demo=]]                                               | Just a demo module for testing and documentation purposes                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/emfit/__init__.py][=my.emfit=]]                                    | [[https://shop-eu.emfit.com/products/emfit-qs][Emfit QS]] sleep tracker                                                                            |
| [[https://github.com/karlicoss/HPI/tree/master/my/endomondo.py][=my.endomondo=]]                                     | Endomondo exercise data                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/fbmessenger.py][=my.fbmessenger=]]                                 | Facebook Messenger messages                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/foursquare.py][=my.foursquare=]]                                   | Foursquare/Swarm checkins                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/github/all.py][=my.github.all=]]                                   | Unified Github data (merged from GDPR export and periodic API updates)                                                                             |
| [[https://github.com/karlicoss/HPI/tree/master/my/github/gdpr.py][=my.github.gdpr=]]                                 | Github data (uses [[https://github.com/settings/admin][official GDPR export]])                                                                     |
| [[https://github.com/karlicoss/HPI/tree/master/my/github/ghexport.py][=my.github.ghexport=]]                         | Github data: events, comments, etc. (API data)                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/hypothesis.py][=my.hypothesis=]]                                   | [[https://hypothes.is][Hypothes.is]] highlights and annotations                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/instapaper.py][=my.instapaper=]]                                   | [[https://www.instapaper.com][Instapaper]] bookmarks, highlights and annotations                                                                   |
| [[https://github.com/karlicoss/HPI/tree/master/my/kobo.py][=my.kobo=]]                                               | [[https://uk.kobobooks.com/products/kobo-aura-one][Kobo]] e-ink reader: annotations and reading stats                                              |
| [[https://github.com/karlicoss/HPI/tree/master/my/lastfm.py][=my.lastfm=]]                                           | Last.fm scrobbles                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/location/google.py][=my.location.google=]]                         | Location data from Google Takeout                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/location/home.py][=my.location.home=]]                             | Simple location provider, serving as a fallback when more detailed data isn't available                                                            |
| [[https://github.com/karlicoss/HPI/tree/master/my/materialistic.py][=my.materialistic=]]                             | [[https://play.google.com/store/apps/details?id=io.github.hidroh.materialistic][Materialistic]] app for Hackernews                                 |
| [[https://github.com/karlicoss/HPI/tree/master/my/orgmode.py][=my.orgmode=]]                                         | Programmatic access and queries to org-mode files on the filesystem                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/pdfs.py][=my.pdfs=]]                                               | PDF documents and annotations on your filesystem                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/photos/main.py][=my.photos.main=]]                                 | Photos and videos on your filesystem, their GPS and timestamps                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/pinboard.py][=my.pinboard=]]                                       | [[https://pinboard.in][Pinboard]] bookmarks                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/pocket.py][=my.pocket=]]                                           | [[https://getpocket.com][Pocket]] bookmarks and highlights                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/polar.py][=my.polar=]]                                             | [[https://github.com/burtonator/polar-bookshelf][Polar]] articles and highlights                                                                   |
| [[https://github.com/karlicoss/HPI/tree/master/my/reddit.py][=my.reddit=]]                                           | Reddit data: saved items/comments/upvotes/etc.                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/rescuetime.py][=my.rescuetime=]]                                   | Rescuetime (phone activity tracking) data.                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/roamresearch.py][=my.roamresearch=]]                               | [[https://roamresearch.com][Roam]] data                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/rss/all.py][=my.rss.all=]]                                         | Unified RSS data, merged from different services I used historically                                                                               |
| [[https://github.com/karlicoss/HPI/tree/master/my/rss/feedbin.py][=my.rss.feedbin=]]                                 | Feedbin RSS reader                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/rss/feedly.py][=my.rss.feedly=]]                                   | Feedly RSS reader                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/rtm.py][=my.rtm=]]                                                 | [[https://rememberthemilk.com][Remember The Milk]] tasks and notes                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/runnerup.py][=my.runnerup=]]                                       | [[https://github.com/jonasoreland/runnerup][Runnerup]] exercise data (TCX format)                                                                  |
| [[https://github.com/karlicoss/HPI/tree/master/my/smscalls.py][=my.smscalls=]]                                       | Phone calls and SMS messages                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/stackexchange/gdpr.py][=my.stackexchange.gdpr=]]                   | Stackexchange data (uses [[https://stackoverflow.com/legal/gdpr/request][official GDPR export]])                                                   |
| [[https://github.com/karlicoss/HPI/tree/master/my/stackexchange/stexport.py][=my.stackexchange.stexport=]]           | Stackexchange data (uses API via [[https://github.com/karlicoss/stexport][stexport]])                                                              |
| [[https://github.com/karlicoss/HPI/tree/master/my/taplog.py][=my.taplog=]]                                           | [[https://play.google.com/store/apps/details?id=com.waterbear.taglog][Taplog]] app data                                                            |
| [[https://github.com/karlicoss/HPI/tree/master/my/time/tz/main.py][=my.time.tz.main=]]                               | Timezone data provider, used to localize timezone-unaware timestamps for other modules                                                             |
| [[https://github.com/karlicoss/HPI/tree/master/my/time/tz/via_location.py][=my.time.tz.via_location=]]               | Timezone data provider, guesses timezone based on location data (e.g. GPS)                                                                         |
| [[https://github.com/karlicoss/HPI/tree/master/my/twitter/all.py][=my.twitter.all=]]                                 | Unified Twitter data (merged from the archive and periodic updates)                                                                                |
| [[https://github.com/karlicoss/HPI/tree/master/my/twitter/archive.py][=my.twitter.archive=]]                         | Twitter data (uses [[https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive][official twitter archive export]])    |
| [[https://github.com/karlicoss/HPI/tree/master/my/twitter/twint.py][=my.twitter.twint=]]                             | Twitter data (tweets and favorites). Uses [[https://github.com/twintproject/twint][Twint]] data export.                                            |
| [[https://github.com/karlicoss/HPI/tree/master/my/vk/vk_messages_backup.py][=my.vk.vk_messages_backup=]]             | VK data (exported by [[https://github.com/Totktonada/vk_messages_backup][Totktonada/vk_messages_backup]])                                          |
:END:

Some modules are private, and need a bit of cleanup before merging:

| my.workouts     | Exercise activity, from Endomondo and manual logs                                |
| my.sleep.manual | Subjective sleep data, manually logged                                           |
| my.nutrition    | Food and drink consumption data, logged manually from different sources          |
| my.money        | Expenses and shopping data                                                       |
| my.webhistory   | Browsing history (part of [[https://github.com/karlicoss/promnesia][promnesia]]) |

#+html:

* How do you use it?
:PROPERTIES:
:CUSTOM_ID: usecases
:END:
Mainly I use it as a data provider for my scripts, tools, and dashboards.

Also, check out [[https://beepb00p.xyz/myinfra.html#mypkg][my infrastructure map]]. It might be helpful for understanding what's my vision on HPI.
** Instant search
:PROPERTIES:
:CUSTOM_ID: search
:END:
Typical search interfaces make me unhappy as they are *siloed, slow, awkward to use and don't work offline*.
So I built my own ways around it! I write about it in detail [[https://beepb00p.xyz/pkm-search.html#personal_information][here]].

In essence, I'm mirroring most of my online data like chat logs, comments, etc., as plaintext.
I can overview it in any text editor, and incrementally search over *all of it* in a single keypress.
** orger
:PROPERTIES:
:CUSTOM_ID: orger
:END:
[[https://github.com/karlicoss/orger][orger]] is a tool that helps you generate an org-mode representation of your data.

It lets you benefit from the existing tooling and infrastructure around org-mode, the most famous being Emacs.

I'm using it for:

- searching, overviewing and navigating the data
- creating tasks straight from the apps (e.g. Reddit/Telegram)
- spaced repetition via [[https://orgmode.org/worg/org-contrib/org-drill.html][org-drill]]

Orger comes with some existing [[https://github.com/karlicoss/orger/tree/master/modules][modules]], but it should be easy to adapt your own data source if you need something else.

I write about it in detail [[http://beepb00p.xyz/orger.html][here]] and [[http://beepb00p.xyz/orger-todos.html][here]].
** promnesia
:PROPERTIES:
:CUSTOM_ID: promnesia
:END:
[[https://github.com/karlicoss/promnesia#demo][promnesia]] is a browser extension I'm working on to escape silos by *unifying annotations and browsing history* from different data sources.

I've been using it for more than a year now and working on final touches to properly release it for other people.
** dashboard
:PROPERTIES:
:CUSTOM_ID: dashboard
:END:

As a big fan of [[https://beepb00p.xyz/tags.html#quantified-self][#quantified-self]], I'm working on personal health, sleep and exercise dashboard, built from various data sources.

I'm working on making it public, you can see some screenshots [[https://www.reddit.com/r/QuantifiedSelf/comments/cokt4f/what_do_you_all_do_with_your_data/ewmucgk][here]].
** timeline
:PROPERTIES:
:CUSTOM_ID: timeline
:END:

Timeline is a [[https://beepb00p.xyz/tags.html#lifelogging][#lifelogging]] project I'm working on.

I want to see all my digital history, search in it, filter, easily jump at a specific point in time and see the context when it happened.
That way it works as a sort of external memory.

Ideally, it would look similar to Andrew Louis's [[https://hyfen.net/memex][Memex]], or might even reuse his interface if
he open sources it. I highly recommend watching his talk for inspiration.

* Ad-hoc and interactive
:PROPERTIES:
:CUSTOM_ID: interactive
:END:

** What were my music listening stats for 2018?
:PROPERTIES:
:CUSTOM_ID: lastfm
:END:

Single import away from getting tracks you listened to:

#+begin_src python
  from my.lastfm import scrobbles
  list(scrobbles())[200: 205]
#+end_src

: [Scrobble(raw={'album': 'Nevermind', 'artist': 'Nirvana', 'date': '1282488504', 'name': 'Drain You'}),
:  Scrobble(raw={'album': 'Dirt', 'artist': 'Alice in Chains', 'date': '1282489764', 'name': 'Would?'}),
:  Scrobble(raw={'album': 'Bob Dylan: The Collection', 'artist': 'Bob Dylan', 'date': '1282493517', 'name': 'Like a Rolling Stone'}),
:  Scrobble(raw={'album': 'Dark Passion Play', 'artist': 'Nightwish', 'date': '1282493819', 'name': 'Amaranth'}),
:  Scrobble(raw={'album': 'Rolled Gold +', 'artist': 'The Rolling Stones', 'date': '1282494161', 'name': "You Can't Always Get What You Want"})]

Or, as a pretty Pandas frame:

#+begin_src python
  import pandas as pd
  df = pd.DataFrame([{
      'dt': s.dt,
      'track': s.track,
  } for s in scrobbles()]).set_index('dt')
  df[200: 205]
#+end_src

:                                                                        track
: dt
: 2010-08-22 14:48:24+00:00                                Nirvana — Drain You
: 2010-08-22 15:09:24+00:00                           Alice in Chains — Would?
: 2010-08-22 16:11:57+00:00                   Bob Dylan — Like a Rolling Stone
: 2010-08-22 16:16:59+00:00                               Nightwish — Amaranth
: 2010-08-22 16:22:41+00:00  The Rolling Stones — You Can't Always Get What...

We can use [[https://github.com/martijnvermaat/calmap][calmap]] library to plot a github-style music listening activity heatmap:

#+begin_src python
  import matplotlib.pyplot as plt
  plt.figure(figsize=(10, 2.3))

  import calmap
  df = df.set_index(df.index.tz_localize(None)) # calmap expects tz-unaware dates
  calmap.yearplot(df['track'], how='count', year=2018)

  plt.tight_layout()
  plt.title('My music listening activity for 2018')
  plot_file = 'hpi_files/lastfm_2018.png'
  plt.savefig(plot_file)
  plot_file
#+end_src

[[https://beepb00p.xyz/hpi_files/lastfm_2018.png]]

This isn't necessarily very insightful data, but fun to look at now and then!

** What are the most interesting Slate Star Codex posts I've read?
:PROPERTIES:
:CUSTOM_ID: hypothesis_stats
:END:

My friend asked me if I could recommend them posts I found interesting on [[https://slatestarcodex.com][Slate Star Codex]].
With few lines of Python I can quickly recommend them posts I engaged most with, i.e. the ones I annotated most on [[https://hypothes.is][Hypothesis]].

#+begin_src python
  from my.hypothesis import pages
  from collections import Counter
  cc = Counter({(p.title + ' ' + p.url): len(p.highlights) for p in pages() if 'slatestarcodex' in p.url})
  return cc.most_common(10)
#+end_src

| The Anti-Reactionary FAQ http://slatestarcodex.com/2013/10/20/the-anti-reactionary-faq/                                                                                | 32 |
| Reactionary Philosophy In An Enormous, Planet-Sized Nutshell https://slatestarcodex.com/2013/03/03/reactionary-philosophy-in-an-enormous-planet-sized-nutshell/                     | 17 |
| The Toxoplasma Of Rage http://slatestarcodex.com/2014/12/17/the-toxoplasma-of-rage/                                                                                | 16 |
| What Universal Human Experiences Are You Missing Without Realizing It? https://slatestarcodex.com/2014/03/17/what-universal-human-experiences-are-you-missing-without-realizing-it/ | 16 |
| Meditations On Moloch http://slatestarcodex.com/2014/07/30/meditations-on-moloch/                                                                                | 12 |
| Universal Love, Said The Cactus Person http://slatestarcodex.com/2015/04/21/universal-love-said-the-cactus-person/                                                                  | 11 |
| Untitled http://slatestarcodex.com/2015/01/01/untitled/                                                                                | 11 |
| Considerations On Cost Disease https://slatestarcodex.com/2017/02/09/considerations-on-cost-disease/                                                                                | 10 |
| In Defense of Psych Treatment for Attempted Suicide http://slatestarcodex.com/2013/04/25/in-defense-of-psych-treatment-for-attempted-suicide/                                       |  9 |
| I Can Tolerate Anything Except The Outgroup https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything-except-the-outgroup/                                                      |  9 |

** Accessing exercise data
:PROPERTIES:
:CUSTOM_ID: exercise
:END:
 E.g. see use of ~my.workouts~ [[https://beepb00p.xyz/heartbeats_vs_kcals.html][here]].

** Book reading progress
:PROPERTIES:
:CUSTOM_ID: kobo_progress
:END:

I publish my reading stats on [[https://www.goodreads.com/user/show/22191391-dima-gerasimov][Goodreads]] so other people can see what I'm reading/have read, but Kobo [[https://beepb00p.xyz/ideas.html#kobo2goodreads][lacks integration]] with Goodreads.
I'm using [[https://github.com/karlicoss/kobuddy][kobuddy]] to access my my Kobo data, and I've got a regular task that reminds me to sync my progress once a month.

The task looks like this:

#+begin_src org
  ,* TODO [#C] sync [[https://goodreads.com][reading progress]] with kobo
    DEADLINE: <2019-11-24 Sun .+4w -0d>
  [[eshell: python3 -c 'import my.kobo; my.kobo.print_progress()']]
#+end_src

With a single Enter keypress on the inlined =eshell:= command I can print the progress and fill in the completed books on Goodreads, e.g.:

#+begin_example

  A_Mathematician's_Apology by G. H. Hardy
  Started : 21 Aug 2018 11:44
  Finished: 22 Aug 2018 12:32

  Fear and Loathing in Las Vegas: A Savage Journey to the Heart of the American Dream (Vintage) by Thompson, Hunter S.
  Started : 06 Sep 2018 05:54
  Finished: 09 Sep 2018 12:21

  Sapiens: A Brief History of Humankind by Yuval Noah Harari
  Started : 09 Sep 2018 12:22
  Finished: 16 Sep 2018 07:25

  Inadequate Equilibria: Where and How Civilizations Get Stuck by Eliezer Yudkowsky
  Started : 31 Jul 2018 22:54
  Finished: 16 Sep 2018 07:25

  Albion Dreaming by Andy Roberts
  Started : 20 Aug 2018 21:16
  Finished: 16 Sep 2018 07:26
#+end_example

** Messenger stats
:PROPERTIES:
:CUSTOM_ID: messenger_stats
:END:
How much do I chat on Facebook Messenger?

#+begin_src python
  from my.fbmessenger import messages

  import pandas as pd
  import matplotlib.pyplot as plt

  df = pd.DataFrame({'dt': m.dt, 'messages': 1} for m in messages())
  df.set_index('dt', inplace=True)

  df = df.resample('M').sum() # by month
  df = df.loc['2016-01-01':'2019-01-01'] # past subset for determinism

  fig, ax = plt.subplots(figsize=(15, 5))
  df.plot(kind='bar', ax=ax)

  # todo wonder if that vvv can be less verbose...
  x_labels = df.index.strftime('%Y %b')
  ax.set_xticklabels(x_labels)

  plot_file = 'hpi_files/messenger_2016_to_2019.png'
  plt.tight_layout()
  plt.savefig(plot_file)
  return plot_file
#+end_src

[[https://beepb00p.xyz/hpi_files/messenger_2016_to_2019.png]]

** Which month in 2020 did I make the most git commits in?
:PROPERTIES:
:CUSTOM_ID: hpi_query_git
:END:

If you like the shell or just want to quickly convert/grab some information from HPI, it also comes with a JSON query interface - so you can export the data, or just pipeline to your heart's content:

#+begin_src bash
  $ hpi query my.coding.commits.commits --stream # stream JSON objects as they're read
    --order-type datetime  # find the 'datetime' attribute and order by that
    --after '2020-01-01' --before '2021-01-01' # in 2020
    | jq '.committed_dt' -r  # extract the datetime
    # mangle the output a bit to group by month and graph it
    | cut -d'-' -f-2 | sort | uniq -c | awk '{print $2,$1}' | sort -n | termgraph
#+end_src

#+begin_src
2020-01: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 458.00
2020-02: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 440.00
2020-03: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 545.00
2020-04: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 585.00
2020-05: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 518.00
2020-06: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 755.00
2020-07: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 467.00
2020-08: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 449.00
2020-09: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 1.03 K
2020-10: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 791.00
2020-11: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 474.00
2020-12: ▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 383.00
#+end_src

See [[https://github.com/karlicoss/HPI/blob/master/doc/QUERY.md][query docs]]
for more examples

** Querying Roam Research database
:PROPERTIES:
:CUSTOM_ID: roamresearch
:END:
I've got some code examples [[https://beepb00p.xyz/myinfra-roam.html#interactive][here]].

* How does it get input data?
:PROPERTIES:
:CUSTOM_ID: input_data
:END:
If you're curious about any specific data sources I'm using, I've written it up [[https://beepb00p.xyz/my-data.html][in detail]].

Also see [[https://github.com/karlicoss/HPI/blob/master/doc/SETUP.org#data-flow]["Data flow"]] documentation with some nice diagrams explaining on specific examples.

In short:

- The data is [[https://beepb00p.xyz/myinfra.html#exports][periodically synchronized]] from the services (cloud or not) locally, on the filesystem

  As a result, you get [[https://beepb00p.xyz/myinfra.html#fs][JSONs/sqlite]] (or other formats, depending on the service) on your disk.

  Once you have it, it's trivial to back it up and synchronize to other computers/phones, if necessary.

  To schedule periodic sync, I'm using [[https://beepb00p.xyz/scheduler.html#cron][cron]].

- =my.= package only accesses the data on the filesystem

  That makes it extremely fast, reliable, and fully offline capable.

As you can see, in such a setup, the data is lagging behind the 'realtime'.
I consider it a necessary sacrifice to make everything fast and resilient.

In theory, it's possible to make the system almost realtime by having a service that sucks in data continuously (rather than periodically), but it's harder as well.

* Q & A
:PROPERTIES:
:CUSTOM_ID: q_and_a
:END:

** Why Python?
:PROPERTIES:
:CUSTOM_ID: why_python
:END:

I don't consider Python unique as a language suitable for such a project.
It just happens to be the one I'm most comfortable with.
I do have some reasons that I think make it /specifically/ good, but explaining them is out of this post's scope.

In addition, Python offers a [[https://github.com/karlicoss/awesome-python#data-analysis][very rich ecosystem]] for data analysis, which we can use to our benefit.

That said, I've never seen anything similar in other programming languages, and I would be really interested in, so please send me links if you know some.
I've heard LISPs are great for data? ;)

Overall, I wish [[https://en.wikipedia.org/wiki/Foreign_function_interface][FFIs]] were a bit more mature, so we didn't have to think about specific programming languages at all.

** Can anyone use it?
:PROPERTIES:
:CUSTOM_ID: can_anyone_use_it
:END:
Yes!

- you can plug in *your own data*
- most modules are isolated, so you can only use the ones that you want to
- everything is easily *extensible*

  Starting from simply adding new modules to any dynamic hackery you can possibly imagine within Python.

** How easy is it to use?
:PROPERTIES:
:CUSTOM_ID: how_easy_to_use
:END:
The whole setup requires some basic programmer literacy:

- installing/running and potentially modifying Python code
- using symlinks
- potentially running Cron jobs

If you have any ideas on making the setup simpler, please let me know!

** What about privacy?
:PROPERTIES:
:CUSTOM_ID: privacy
:END:
The modules contain *no data, only code* to operate on the data.

Everything is [[https://beepb00p.xyz/tags.html#offline][*local first*]], the input data is on your filesystem.
If you're truly paranoid, you can even wrap it in a Docker container.

There is still a question of whether you trust yourself at even keeping all the data on your disk, but it is out of the scope of this post.

If you'd rather keep some code private too, it's also trivial to achieve with a private subpackage.

** But /should/ I use it?
:PROPERTIES:
:CUSTOM_ID: should_i_use_it
:END:

#+begin_quote
Sure, maybe you can achieve a perfect system where you can instantly find and recall anything that you've done. Do you really want it?
Wouldn't that, like, make you less human?
#+end_quote

I'm not a gatekeeper of what it means to be human, but I don't think that the shortcomings of the human brain are what makes us such.

So I can't answer that for you. I certainly want it though.
I'm [[https://beepb00p.xyz/tags.html#pkm][quite open]] about my goals -- I'd happily get merged/augmented with a computer to enhance my thinking and analytical abilities.

While at the moment [[https://en.wikipedia.org/wiki/Hard_problem_of_consciousness][we don't even remotely understand]] what would such merging or "mind uploading" entail exactly,
I can clearly delegate some tasks, like long term memory, information lookup, and data processing to a computer. They can already handle it really well.

#+begin_quote
What about these people who have perfect recall and wish they hadn't.
#+end_quote

Sure, maybe it sucks. At the moment though, my recall is far from perfect, and this only annoys me.
I want to have a choice at least, and digital tools give me this choice.

** Would it suit /me/?
:PROPERTIES:
:CUSTOM_ID: would_it_suit_me
:END:

Probably, at least to some extent.

First, our lives are different, so our APIs might be different too.
This is more of a demonstration of what's I'm using, although I did spend effort towards making it as modular and extensible as possible, so other people could use it too.
It's easy to modify code, add extra methods and modules. You can even keep all your modifications private.

But after all, we've all sharing many similar activities and using the same products, so there is a huge overlap.
I'm not sure how far we can stretch it and keep modules generic enough to be used by multiple people. But let's give it a try perhaps? :)

Second, interacting with your data through the code is the central idea of the project.
That kind of cuts off people without technical skills, and even many people capable of coding,
who dislike the idea of writing code outside of work.

It might be possible to expose some [[https://en.wikipedia.org/wiki/No-code_development_platform][no-code]] interfaces,
but I still feel that wouldn't be enough.

I'm not sure whether it's a solvable problem at this point, but happy to hear any suggestions!

** What it isn't?
:PROPERTIES:
:CUSTOM_ID: what_it_isnt
:END:

- It's not vaporware

  The project is a little crude, but it's real and working. I've been using it for a long time now, and find it fairly sustainable to keep using for the foreseeable future.

- It's not going to be another silo

  While I don't have anything against commercial use (and I believe any work in this area will benefit all of us), I'm not planning to build a product out of it.

  I really hope it can grow into or inspire some mature open source system.

  Please take my ideas and code and build something cool from it!

* HPI Repositories
:PROPERTIES:
:CUSTOM_ID: hpi_repos
:END:

One of HPI's core goals is to be as extendable as possible. The goal here isn't to become a monorepo and support every possible data source/website to the point that this isn't maintainable anymore, but hopefully you get a few modules 'for free'.

If you want to write modules for personal use but don't want to merge them into here, you're free to maintain modules locally in a separate directory to avoid any merge conflicts, and entire HPI repositories can even be published separately and installed into the single ~my~ python package (For more info on this, see [[https://github.com/karlicoss/HPI/tree/master/doc/MODULE_DESIGN.org][MODULE_DESIGN]])

Other HPI Repositories:

- [[https://github.com/seanbreckenridge/HPI][seanbreckenridge/HPI]]
- [[https://github.com/madelinecameron/hpi][madelinecameron/HPI]]

If you want to create your own to create your own modules/override something here, you can use the [[https://github.com/seanbreckenridge/HPI-template][template]].

* Related links
:PROPERTIES:
:CUSTOM_ID: links
:END:
Similar projects:

- [[https://hyfen.net/memex][Memex]] by Andrew Louis
- [[https://github.com/novoid/Memacs][Memacs]] by Karl Voit
- [[https://news.ycombinator.com/item?id=9615901][Me API - turn yourself into an open API (HN)]]
- [[https://github.com/markwk/qs_ledger][QS ledger]] from Mark Koester
- [[https://dogsheep.github.io][Dogsheep]]: a collection of tools for personal analytics using SQLite and Datasette
- [[https://github.com/tehmantra/my][tehmantra/my]]: directly inspired by this package
- [[https://github.com/bcongdon/bolero][bcongdon/bolero]]: exposes your personal data as a REST API
- [[https://en.wikipedia.org/wiki/Solid_(web_decentralization_project)#Design][Solid project]]: personal data pod, which websites pull data from
- [[https://remotestorage.io][remoteStorage]]: open protocol for apps to write data to your own storage
- [[https://perkeep.org]][Perkeep]: a tool with [[https://perkeep.org/doc/principles]][principles] and esp. [[https://perkeep.org/doc/uses]][use cases] for self-sovereign storage of personal data
- [[https://www.openhumans.org]][Open Humans]: a community and infrastructure to analyse and share personal data

Other links:

- NetOpWibby: [[https://news.ycombinator.com/item?id=21684949][A Personal API (HN)]]
- [[https://beepb00p.xyz/sad-infra.html][The sad state of personal data and infrastructure]]: here I am going into motivation and difficulties arising in the implementation
- [[https://beepb00p.xyz/myinfra-roam.html][Extending my personal infrastructure]]: a followup, where I'm demonstrating how to integrate a new data source (Roam Research)

* --
:PROPERTIES:
:CUSTOM_ID: fin
:END:

Open to any feedback and thoughts!

Also, don't hesitate to raise an issue, or reach me personally if you want to try using it, and find the instructions confusing. Your questions would help me to make it simpler!

In some near future I will write more about:

- specific technical decisions and patterns
- challenges I had so solve
- more use-cases and demos -- it's impossible to fit everything in one post!

, but happy to answer any questions on these topics now!


[![Build Status](https://travis-ci.org/nilmtk/nilmtk.svg?branch=master)](https://travis-ci.org/nilmtk/nilmtk) [![Install with conda](https://anaconda.org/nilmtk/nilmtk/badges/installer/conda.svg)](https://anaconda.org/nilmtk/nilmtk) [![conda package version](https://anaconda.org/nilmtk/nilmtk/badges/version.svg)](https://anaconda.org/nilmtk/nilmtk)

# NILMTK: Non-Intrusive Load Monitoring Toolkit

Non-Intrusive Load Monitoring (NILM) is the process of estimating the
energy consumed by individual appliances given just a whole-house
power meter reading.  In other words, it produces an (estimated)
itemised energy bill from just a single, whole-house power meter.

NILMTK is a toolkit designed to help **researchers** evaluate the accuracy of NILM algorithms. If you are a new Python user, it is recommended to educate yourself on [Pandas](https://pandas.pydata.org/), [Pytables](http://www.pytables.org/) and other tools from the Python ecosystem.

**⚠It may take time for the NILMTK authors to get back to you regarding queries/issues. However, you are more than welcome to propose changes, support!** Remember to check existing issue tickets, especially the open ones.

# Documentation

[NILMTK Documentation](https://github.com/nilmtk/nilmtk/tree/master/docs/manual)

If you are a new user, read the [install instructions here](https://github.com/nilmtk/nilmtk/blob/master/docs/manual/user_guide/install_user.md). It came to our attention that some users follow third-party tutorials to install NILMTK. Always remember to check the dates of such tutorials, many are very outdated and don't reflect NILMTK's current version or the recommended/supported setup.

# Why a toolkit for NILM?

We quote our [NILMTK paper](http://arxiv.org/pdf/1404.3878v1.pdf)
explaining the need for a NILM toolkit:

  > Empirically comparing disaggregation algorithms is currently
  > virtually impossible. This is due to the different data sets used,
  > the lack of reference implementations of these algorithms and the
  > variety of accuracy metrics employed.

# What NILMTK provides

To address this challenge, we present the Non-intrusive Load Monitoring
Toolkit (NILMTK); an open source toolkit designed specifically to enable
the comparison of energy disaggregation algorithms in a reproducible
manner. This work is the first research to compare multiple
disaggregation approaches across multiple publicly available data sets.
NILMTK includes:

-  parsers for a range of existing data sets (8 and counting)
-  a collection of preprocessing algorithms
-  a set of statistics for describing data sets
-  a number of [reference benchmark disaggregation algorithms](https://github.com/nilmtk/nilmtk/wiki/NILM-Algorithms)
-  a common set of accuracy metrics
-  and much more!

# Publications

If you use NILMTK in academic work then please consider citing our papers. Here are some of the publications (contributors, please update this as required):

1. Nipun Batra, Jack Kelly, Oliver Parson, Haimonti Dutta, William Knottenbelt, Alex Rogers, Amarjeet Singh, Mani Srivastava. NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring. In: 5th International Conference on Future Energy Systems (ACM e-Energy), Cambridge, UK. 2014. DOI:[10.1145/2602044.2602051](http://dx.doi.org/10.1145/2602044.2602051). arXiv:[1404.3878](http://arxiv.org/abs/1404.3878).
2. Nipun Batra, Jack Kelly, Oliver Parson, Haimonti Dutta, William Knottenbelt, Alex Rogers, Amarjeet Singh, Mani Srivastava. NILMTK: An Open Source Toolkit for Non-intrusive Load Monitoring". In: NILM Workshop, Austin, US. 2014 \[[pdf](http://nilmworkshop14.files.wordpress.com/2014/05/batra_nilmtk.pdf)\]
3. Jack Kelly, Nipun Batra, Oliver Parson, Haimonti Dutta, William Knottenbelt, Alex Rogers, Amarjeet Singh, Mani Srivastava. Demo Abstract: NILMTK v0.2: A Non-intrusive Load Monitoring Toolkit for Large Scale Data Sets. In the first ACM Workshop On Embedded Systems For Energy-Efficient Buildings, 2014. DOI:[10.1145/2674061.2675024](http://dx.doi.org/10.1145/2674061.2675024). arXiv:[1409.5908](http://arxiv.org/abs/1409.5908).
4. Nipun Batra, Rithwik Kukunuri, Ayush Pandey, Raktim Malakar, Rajat Kumar, Odysseas Krystalakos, Mingjun Zhong, Paulo Meira, and Oliver Parson. 2019. Towards reproducible state-of-the-art energy disaggregation. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation (BuildSys '19). Association for Computing Machinery, New York, NY, USA, 193–202. DOI:[10.1145/3360322.3360844](https://doi.org/10.1145/3360322.3360844)

Please note that NILMTK has evolved *a lot* since most of these papers were published! Please use the [online docs](https://github.com/nilmtk/nilmtk/tree/master/docs/manual)
as a guide to the current API.

# Brief history

* August 2019: v0.4 released with the new API. See also [NILMTK-Contrib](https://github.com/nilmtk/nilmtk-contrib).
* June 2019: v0.3.1 released on [Anaconda Cloud](https://anaconda.org/nilmtk/nilmtk/).
* Jav 2018: Initial Python 3 support on the v0.3 branch
* Nov 2014: NILMTK wins best demo award at [ACM BuildSys](http://www.buildsys.org/2014/)
* July 2014: v0.2 released
* June 2014: NILMTK presented at [ACM e-Energy](http://conferences.sigcomm.org/eenergy/2014/)
* April 2014: v0.1 released

For more detail, please see our [changelog](https://github.com/nilmtk/nilmtk/blob/master/docs/manual/development_guide/changelog.md).


# OmegaConf
|  | Description |
| --- | --- |
| Project | [![PyPI version](https://badge.fury.io/py/omegaconf.svg)](https://badge.fury.io/py/omegaconf)[![Downloads](https://pepy.tech/badge/omegaconf/month)](https://pepy.tech/project/omegaconf?versions=1.4.*&versions=2.0.*&versions=2.1.*)![PyPI - Python Version](https://img.shields.io/pypi/pyversions/omegaconf.svg) |
| Code quality| [![CircleCI](https://img.shields.io/circleci/build/github/omry/omegaconf?logo=s&token=5de2f8dc2a0dd78438520575431aa533150806e3)](https://circleci.com/gh/omry/omegaconf)[![Coverage Status](https://coveralls.io/repos/github/omry/omegaconf/badge.svg)](https://coveralls.io/github/omry/omegaconf)[![Total alerts](https://img.shields.io/lgtm/alerts/g/omry/omegaconf.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/omry/omegaconf/alerts/)[![Language grade: Python](https://img.shields.io/lgtm/grade/python/g/omry/omegaconf.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/omry/omegaconf/context:python)|
| Docs and support |[![Documentation Status](https://readthedocs.org/projects/omegaconf/badge/?version=2.0_branch)](https://omegaconf.readthedocs.io/en/2.1_branch/)[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/omry/omegaconf/master?filepath=docs%2Fnotebook%2FTutorial.ipynb)|

OmegaConf is a hierarchical configuration system, with support for merging configurations from multiple sources (YAML config files, dataclasses/objects and CLI arguments)
providing a consistent API regardless of how the configuration was created.

## Releases

### Stable (2.3)
OmegaConf 2.3 is the current stable version.
* [What's new](https://github.com/omry/omegaconf/releases/tag/v2.3.0)
* [Documentation](https://omegaconf.readthedocs.io/en/2.3_branch/)
* [Source code](https://github.com/omry/omegaconf/tree/2.3_branch)

Install with `pip install --upgrade omegaconf`

### Previous release (2.2)

* [What's new](https://github.com/omry/omegaconf/releases/tag/v2.1.1)
* [Documentation](https://omegaconf.readthedocs.io/en/2.1_branch/)
* [Slides](https://docs.google.com/presentation/d/e/2PACX-1vT_UIV7hCnquIbLUm4NnkUpXvPEh33IKiUEvPRF850WKA8opOlZOszjKdZ3tPmf8u7hGNP6HpqS-NT5/pub?start=false&loop=false&delayms=3000)
* [Source code](https://github.com/omry/omegaconf/tree/2.1_branch)

Install with `pip install omegaconf==2.1`

### Previous release (2.1)

* [What's new](https://github.com/omry/omegaconf/releases/tag/v2.1.1)
* [Documentation](https://omegaconf.readthedocs.io/en/2.1_branch/)
* [Slides](https://docs.google.com/presentation/d/e/2PACX-1vT_UIV7hCnquIbLUm4NnkUpXvPEh33IKiUEvPRF850WKA8opOlZOszjKdZ3tPmf8u7hGNP6HpqS-NT5/pub?start=false&loop=false&delayms=3000)
* [Source code](https://github.com/omry/omegaconf/tree/2.1_branch)

Install with `pip install omegaconf==2.1`

### Previous release (2.0)

* [What's new](https://github.com/omry/omegaconf/releases/tag/v2.0.0)
* [Documentation](https://omegaconf.readthedocs.io/en/2.0_branch/)
* [Slides](https://docs.google.com/presentation/d/e/2PACX-1vT_UIV7hCnquIbLUm4NnkUpXvPEh33IKiUEvPRF850WKA8opOlZOszjKdZ3tPmf8u7hGNP6HpqS-NT5/pub?start=false&loop=false&delayms=3000)
* [Source code](https://github.com/omry/omegaconf/tree/2.0_branch)

Install with `pip install omegaconf==2.0.6`

## Live tutorial
Run the live tutorial: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/omry/omegaconf/master?filepath=docs%2Fnotebook%2FTutorial.ipynb)


![](https://raw.githubusercontent.com/textX/textX/master/art/textX-logo.png)

[![PyPI Version](https://img.shields.io/pypi/v/textX.svg)](https://pypi.python.org/pypi/textX)
![](https://img.shields.io/pypi/l/textX.svg)
[![Build status](https://github.com/textx/textx/actions/workflows/tests.yml/badge.svg)](https://github.com/textx/textx/actions)
[![Code test coverage](https://coveralls.io/repos/github/textX/textX/badge.svg?branch=master)](https://coveralls.io/github/textX/textX?branch=master)
[![Documentation Status](https://img.shields.io/badge/docs-latest-green.svg)](http://textx.github.io/textX/latest/)

textX is a meta-language for building Domain-Specific Languages (DSLs) in
Python. It is inspired by [Xtext].

In a nutshell, textX will help you build your textual language in an easy way.
You can invent your own language or build a support for already existing textual
language or file format.

From a single language description (grammar), textX will build a parser and a
meta-model (a.k.a. abstract syntax) for the language. See the docs for the
details.

textX follows the syntax and semantics of Xtext but [differs in some
places](http://textx.github.io/textX/latest/about/comparison/) and is
implemented 100% in Python using [Arpeggio] PEG parser - no grammar ambiguities,
unlimited lookahead, interpreter style of work.

## Quick intro

Here is a complete example that shows the definition of a simple DSL for
drawing. We also show how to define a custom class, interpret models and search
for instances of a particular type.

```python
from textx import metamodel_from_str, get_children_of_type

grammar = """
Model: commands*=DrawCommand;
DrawCommand: MoveCommand | ShapeCommand;
ShapeCommand: LineTo | Circle;
MoveCommand: MoveTo | MoveBy;
MoveTo: 'move' 'to' position=Point;
MoveBy: 'move' 'by' vector=Point;
Circle: 'circle' radius=INT;
LineTo: 'line' 'to' point=Point;
Point: x=INT ',' y=INT;
"""

# We will provide our class for Point.
# Classes for other rules will be dynamically generated.
class Point:
    def __init__(self, parent, x, y):
        self.parent = parent
        self.x = x
        self.y = y

    def __str__(self):
        return "{},{}".format(self.x, self.y)

    def __add__(self, other):
        return Point(self.parent, self.x + other.x, self.y + other.y)

# Create meta-model from the grammar. Provide `Point` class to be used for
# the rule `Point` from the grammar.
mm = metamodel_from_str(grammar, classes=[Point])

model_str = """
    move to 5, 10
    line to 10, 10
    line to 20, 20
    move by 5, -7
    circle 10
    line to 10, 10
"""

# Meta-model knows how to parse and instantiate models.
model = mm.model_from_str(model_str)

# At this point model is a plain Python object graph with instances of
# dynamically created classes and attributes following the grammar.

def cname(o):
    return o.__class__.__name__

# Let's interpret the model
position = Point(None, 0, 0)
for command in model.commands:
    if cname(command) == 'MoveTo':
        print('Moving to position', command.position)
        position = command.position
    elif cname(command) == 'MoveBy':
        position = position + command.vector
        print('Moving by', command.vector, 'to a new position', position)
    elif cname(command) == 'Circle':
        print('Drawing circle at', position, 'with radius', command.radius)
    else:
        print('Drawing line from', position, 'to', command.point)
        position = command.point
print('End position is', position)

# Output:
# Moving to position 5,10
# Drawing line from 5,10 to 10,10
# Drawing line from 10,10 to 20,20
# Moving by 5,-7 to a new position 25,13
# Drawing circle at 25,13 with radius 10
# Drawing line from 25,13 to 10,10

# Collect all points starting from the root of the model
points = get_children_of_type("Point", model)
for point in points:
    print('Point: {}'.format(point))

# Output:
# Point: 5,10
# Point: 10,10
# Point: 20,20
# Point: 5,-7
# Point: 10,10
```

## Video tutorials

### Introduction to textX

[![Introduction to
textX](https://img.youtube.com/vi/CN2IVtInapo/0.jpg)](https://www.youtube.com/watch?v=CN2IVtInapo)

### Implementing Martin Fowler's State Machine DSL in textX

[![Implementing State Machine
DSL](https://img.youtube.com/vi/HI14jk0JIR0/0.jpg)](https://www.youtube.com/watch?v=HI14jk0JIR0)

## Docs and tutorials

The full documentation with tutorials is available at
http://textx.github.io/textX/stable/

You can also try textX in [our
playground](https://textx.github.io/textx-playground/). There is a dropdown with
several examples to get you started.

# Support in IDE/editors

Projects that are currently in progress are:

- [textX-LS](https://github.com/textX/textX-LS) - support for Language Server
  Protocol and VS Code for any textX based language. This project is about to
  supersede the following projects:
  - [textX-languageserver](https://github.com/textX/textX-languageserver) -
    Language Server Protocol support for textX languages
  - [textX-extensions](https://github.com/textX/textX-extensions) - syntax
    highlighting, code outline
- [viewX](https://github.com/danielkupco/viewX-vscode) - creating visualizers
  for textX languages

If you are a vim editor user check
out [support for vim](https://github.com/textX/textx.vim/).

For emacs there is [textx-mode](https://github.com/textX/textx-mode) which is
also available in [MELPA](https://melpa.org/#/textx-mode).

You can also check
out [textX-ninja project](https://github.com/textX/textX-ninja). It is
currently unmaintained.

## Discussion and help

For general questions, suggestions, and feature requests please use [GitHub
Discussions](https://github.com/textX/textX/discussions).

For issues please use [GitHub issue
tracker](https://github.com/textX/textX/issues).

## Citing textX

If you are using textX in your research project we would be very grateful if you
cite our paper:

Dejanović I., Vaderna R., Milosavljević G., Vuković Ž. (2017). [TextX: A Python
tool for Domain-Specific Languages
implementation](https://www.doi.org/10.1016/j.knosys.2016.10.023).
Knowledge-Based Systems, 115, 1-4.

## License

MIT

## Python versions

Tested for 3.8+

[Arpeggio]: https://github.com/textX/Arpeggio
[Xtext]: http://www.eclipse.org/Xtext/


.. image:: https://travis-ci.org/sfepy/sfepy.svg?branch=master
    :target: https://travis-ci.org/sfepy/sfepy
    :alt: CI

========================================
SfePy (Simple finite elements in Python)
========================================

SfePy [1]_ is a software for solving systems of coupled partial differential
equations (PDEs) by the finite element method in 1D, 2D and 3D. It can be
viewed both as black-box PDE solver, and as a Python package which can be used
for building custom applications. The word "simple" means that complex FEM
problems can be coded very easily and rapidly. Its source code can be found on
github [2]_.

SfePy is based primarily on NumPy [3]_ and SciPy [4]_. See the INSTALL file for
installation instructions or refer to [1]_.

- License: New BSD License, see the LICENSE file.

- Authors: Robert Cimrman and Contributors, see the AUTHORS file.

How to start
------------

SfePy uses so called "problem definition files" (also referred to as "input
files" or "problem description files"") that describe the partial differential
equations (PDEs), boundary conditions, function spaces and other ingredients of
the finite element (FE) formulation of a PDE-related problem, see [1]_. The
PDEs are given in weak formulation as usual in the FE context, see [5]_, where
each equation is composed of one or more terms. To see which terms are
available consult ``doc/sfepy_manual.pdf``, or see them online [6]_.

In order to solve a problem, a problem description file has to be created.
There is also an interactive solution support for advanced users [7]_. When
starting to solve a new problem, it is best to have a look at example problem
definition files in the ``sfepy/examples/`` directory - copy the one that is
similar to the problem at hand, and modify it.

While a problem definition file describes a mathematical problem, it does not
contain a discretized solution domain (a FE mesh). The FE mesh must be provided
in another file in one of the supported formats, notably the legacy VTK format
[8]_. SfePy does not provide meshing tools, but it can use a number of standard
formats. The results are almost exclusively stored in legacy VTK files, or
custom HDF5 files. Many standard open-source tools can be used to display the
VTK files, namely paraview [9]_ or pyvista [10]_. The latter is supported
directly within SfePy.

Once an input file and a corresponding mesh file are prepared, the solution of
the problem can be sought, see the documentation.

References
----------

.. [1] https://sfepy.org - main SfePy development site (releases, mailing lists,
       wiki, issue tracking, downloads, documentation, examples)
.. [2] https://github.com/sfepy - master git repository
.. [3] http://numpy.org
.. [4] http://scipy.org
.. [5] http://en.wikipedia.org/wiki/Weak_formulation
.. [6] https://docs.sfepy.org/doc/terms_overview.html
.. [7] https://docs.sfepy.org/doc/tutorial.html#interactive-example-linear-elasticity
.. [8] http://www.vtk.org/VTK/img/file-formats.pdf
.. [9] http://paraview.org/
.. [10] https://docs.pyvista.org/


# IPython Kernel for Jupyter

[![Build Status](https://github.com/ipython/ipykernel/actions/workflows/ci.yml/badge.svg?query=branch%3Amain++)](https://github.com/ipython/ipykernel/actions/workflows/ci.yml/badge.svg?query=branch%3Amain++)
[![Documentation Status](https://readthedocs.org/projects/ipykernel/badge/?version=latest)](http://ipykernel.readthedocs.io/en/latest/?badge=latest)

This package provides the IPython kernel for Jupyter.

## Installation from source

1. `git clone`
1. `cd ipykernel`
1. `pip install -e ".[test]"`

After that, all normal `ipython` commands will use this newly-installed version of the kernel.

## Running tests

Follow the instructions from `Installation from source`.

and then from the root directory

```bash
pytest
```

## Running tests with coverage

Follow the instructions from `Installation from source`.

and then from the root directory

```bash
pytest -vv -s --cov ipykernel --cov-branch --cov-report term-missing:skip-covered --durations 10
```

## About the IPython Development Team

The IPython Development Team is the set of all contributors to the IPython project.
This includes all of the IPython subprojects.

The core team that coordinates development on GitHub can be found here:
https://github.com/ipython/.

## Our Copyright Policy

IPython uses a shared copyright model. Each contributor maintains copyright
over their contributions to IPython. But, it is important to note that these
contributions are typically only changes to the repositories. Thus, the IPython
source code, in its entirety is not the copyright of any single person or
institution. Instead, it is the collective copyright of the entire IPython
Development Team. If individual contributors want to maintain a record of what
changes/contributions they have specific copyright on, they should indicate
their copyright in the commit message of the change, when they commit the
change to one of the IPython repositories.

With this in mind, the following banner should be used in any source code file
to indicate the copyright and license terms:

```
# Copyright (c) IPython Development Team.
# Distributed under the terms of the Modified BSD License.
```


[![Code Testing](https://github.com/scikit-rf/scikit-rf/workflows/Code%20linting%20and%20testing/badge.svg)](https://github.com/scikit-rf/scikit-rf/actions?query=workflow%3A%22Code+linting+and+testing%22)
[![Documentation Status](https://readthedocs.org/projects/scikit-rf/badge/?version=latest)](https://readthedocs.org/projects/scikit-rf/?badge=latest)
[![Coverage Status](https://coveralls.io/repos/scikit-rf/scikit-rf/badge.png)](https://coveralls.io/r/scikit-rf/scikit-rf)
[![License](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](./LICENSE.md)

![PyPI - Downloads](https://img.shields.io/pypi/dm/scikit-rf)
[![Conda Downloads](https://anaconda.org/conda-forge/scikit-rf/badges/downloads.svg)](https://anaconda.org/conda-forge/scikit-rf)

## Description

![scikit-rf_logo](doc/source/_static/scikit-rf-title-flat.png)

**scikit-rf** (aka `skrf`) is an Open Source, BSD-licensed package for RF/Microwave engineering implemented
in the Python programming language. It provides a modern, object-oriented library which is both
flexible and scalable.

* [Home Page](http://www.scikit-rf.org)
* [Documentation](http://scikit-rf.readthedocs.org/)

## Install

[![PyPI version](https://img.shields.io/pypi/v/scikit-rf?style=flat&logo=pypi)](https://pypi.org/project/scikit-rf/)
[![Conda-forge version](https://img.shields.io/conda/v/conda-forge/scikit-rf?style=flat&logo=anaconda)](https://img.shields.io/conda/v/conda-forge/scikit-rf)
[![Conda version](https://anaconda.org/conda-forge/scikit-rf/badges/latest_release_date.svg)](https://anaconda.org/conda-forge/scikit-rf)

If you have [installed Python](https://wiki.python.org/moin/BeginnersGuide/Download), you can install scikit-rf from [pip](https://pypi.org/project/pip/) via
```Shell
python -m pip install scikit-rf
```

If you have [installed conda](https://docs.conda.io/projects/conda/en/latest/user-guide/install/index.html), then you can also get scikit-rf from
```Shell
conda install -c conda-forge scikit-rf
```

**scikit-rf** includes standalone gui applications which can be installed and used like this
```Shell
# Install the additional dependencies for the gui applications
python -m pip install scikit-rf[plot,visa]
# Run the apps
data-grabber # Run the data-grabber application
multiline-trl # Run the multiline-trl application
```

## Questions or comments? Ask the Community

[![Slack](https://img.shields.io/badge/Slack-join%20chat-blueviolet?style=flat&logo=slack)](https://join.slack.com/t/scikit-rf/shared_invite/zt-d82b62wg-0bdSJjZVhHBKf6687V80Jg)
[![Matrix](https://img.shields.io/badge/Matrix-join%20chat-blueviolet?style=flat&logo=matrix)](https://app.element.io/#/room/#scikit-rf:matrix.org)
[![LinkedIn](https://img.shields.io/badge/LinkedIn_scikit_rf_group-orange?style=flat)](https://www.linkedin.com/groups/12462155/)

* [LinkedIn scikit-rf Users Group](https://www.linkedin.com/groups/12462155/): User group to ask and answer questions and to received scikit-rf announcements.
* Chat: scikit-rf have channels on both [Slack](https://join.slack.com/t/scikit-rf/shared_invite/zt-d82b62wg-0bdSJjZVhHBKf6687V80Jg) and [Matrix](https://app.element.io/#/room/#scikit-rf:matrix.org)
* [Suggestion box](https://docs.google.com/forms/d/e/1FAIpQLSfkSNy8KzoHwdDxYGCmHHiA3GsHgN6zKTFI2NrcUR29U7Xx6A/viewform?usp=sf_link): if you would like to suggest a feature/topic for consideration (optionally anonymous).

## Contributing to the scikit-rf project

There is multiple ways to contribute to an open-source project like scikit-rf!

* [Sponsoring the project](https://scikit-rf.readthedocs.io/en/latest/contributing/index.html#sponsoring-the-project)
* [Contributing to the code](https://scikit-rf.readthedocs.io/en/latest/contributing/index.html#contributing-to-the-code)
* [Contributing to the Documentation](https://scikit-rf.readthedocs.io/en/latest/contributing/index.html#contributing-to-the-documentation)
* [Getting some merchandise to express your love](https://scikit-rf.org/merch.html)

![skrfshirt](https://raw.githubusercontent.com/scikit-rf/scikit-rf/master/logo/skrfshirtwhite.png)

## License

scikit-rf is permissively licensed under a [3-clause BSD license](LICENSE.txt).


  pyroSAR

A Python Framework for Large-Scale SAR Satellite Data Processing

The pyroSAR package aims at providing a complete solution for the scalable organization and processing of SAR satellite data:
* Reading of data from various past and present satellite missions
* Handling of acquisition metadata
* User-friendly access to processing utilities in [SNAP](https://step.esa.int/main/toolboxes/snap/)
and [GAMMA Remote Sensing](https://www.gamma-rs.ch/) software
* Formatting of the preprocessed data for further analysis
* Export to Data Cube solutions

Head on over to [readthedocs](https://pyrosar.readthedocs.io/en/latest/?badge=latest) for installation instructions,
examples and API reference.


![FontGoggles Icon](docs/images/icon.png)

# FontGoggles

_Interactive Previewing and Comparing_

For end-user documentation see [fontgoggles.org](https://fontgoggles.org/).

The lastest release can be downloaded [here](https://github.com/justvanrossum/fontgoggles/releases/latest).

FontGoggles was funded by [Google Fonts](https://fonts.google.com/).

## Brief description

FontGoggles is a font viewer desktop app for macOS that supports these file formats:

- `.otf`, `.ttf`, `.otc`, `.ttc`, `.woff`, `.woff2`
- `.ufo`
- `.designspace`
- `.ttx`

It uses HarfBuzz for line layout, so it supports complex scripts. The source formats are partially compiled on the fly, so their line layout and variation behaviour can be realistically previewed.

When a font file changes on disk, FontGoggles will automatically reload it and show the updated version.

## Build instructions

If you want to build and run the app from the source, follow these instructions.

- Make sure you have Python 3.7 installed, preferably from python.org, but homebrew may work, too. (3.8 may work, but I didn't check whether all our dependencies are available.)

- Clone this repository.

- `cd` into the repository folder.

- Setup a virtual environment:

	`$ python3.7 -m venv venv --prompt=fontgoggles`

- Activate the environment:

	`$ source venv/bin/activate`

- Update `pip`:

	`$ pip install --upgrade pip`

- Install dependencies:

	`$ pip install -r requirements.txt`

- Install dev dependencies:

	`$ pip install -r requirements-dev.txt`

- Install our lib:

	`$ pip install .`

- Or, if you prefer an editable install of our lib:

	```
	$ pip install -e .
	$ ./Turbo/build_lib.sh
	```

    (The latter step builds a required C library, that otherwise wouldn't get built in editable mode.)

- Run some tests:

	`$ pytest`

- Build the application:

	`$ python App/setup.py py2app`

You'll find the built application in `App/dist/`

Now you can drop some fonts onto the app, or a folder containing fonts.


# OWSLib

[![Build](https://github.com/geopython/OWSLib/workflows/main.yml/badge.svg)](https://github.com/geopython/OWSLib/actions)
[![License](https://img.shields.io/github/license/geopython/OWSLib.svg)](https://github.com/geopython/OWSLib/blob/master/LICENSE)
[![Chat](https://badges.gitter.im/geopython/OWSLib.svg)](https://app.gitter.im/#/room/#geopython_owslib:gitter.im)

## Overview

OWSLib is a Python package for client programming with Open Geospatial
Consortium (OGC) web service (hence OWS) interface standards, and their
related content models.

Full documentation is available at https://owslib.readthedocs.io

OWSLib provides a common API for accessing service metadata and wrappers for
numerous OGC Web Service interfaces.

## Installation

The easiest way to install pywis-pubsub is via the Python [pip](https://pip.pypa.io)
utility:

```bash
pip3 install OWSLib
```

## Requirements

- Python 3
- [virtualenv](https://virtualenv.pypa.io)

### Dependencies

Dependencies are listed in [requirements.txt](requirements.txt). Dependencies
are automatically installed during OWSLib installation.

### Installing OWSLib

```bash
# setup virtualenv
python3 -m venv owslib
cd owslib
source bin/activate

# clone codebase and install
git clone https://github.com/geopython/OWSLib.git
cd OWSLib
python3 setup.py install
```

## Running

Find out what a WMS has to offer. Service metadata:

```python

>>> from owslib.wms import WebMapService
>>> wms = WebMapService('http://wms.jpl.nasa.gov/wms.cgi', version='1.1.1')
>>> wms.identification.type
'OGC:WMS'
>>> wms.identification.version
'1.1.1'
>>> wms.identification.title
'JPL Global Imagery Service'
>>> wms.identification.abstract
'WMS Server maintained by JPL, worldwide satellite imagery.'

Available layers::

    >>> list(wms.contents)
    ['us_landsat_wgs84', 'modis', 'global_mosaic_base', 'huemapped_srtm',
    'srtm_mag', 'daily_terra', 'us_ned', 'us_elevation', 'global_mosaic',
    'daily_terra_ndvi', 'daily_aqua_ndvi', 'daily_aqua_721', 'daily_planet',
    'BMNG', 'srtmplus', 'us_colordem', None, 'daily_aqua', 'worldwind_dem',
    'daily_terra_721']

Details of a layer::

    >>> wms['global_mosaic'].title
    'WMS Global Mosaic, pan sharpened'
    >>> wms['global_mosaic'].boundingBoxWGS84
    (-180.0, -60.0, 180.0, 84.0)
    >>> wms['global_mosaic'].crsOptions
    ['EPSG:4326', 'AUTO:42003']
    >>> wms['global_mosaic'].styles
    {'pseudo_bright': {'title': 'Pseudo-color image (Uses IR and Visual bands,
    542 mapping), gamma 1.5'}, 'pseudo': {'title': '(default) Pseudo-color
    image, pan sharpened (Uses IR and Visual bands, 542 mapping), gamma 1.5'},
    'visual': {'title': 'Real-color image, pan sharpened (Uses the visual
    bands, 321 mapping), gamma 1.5'}, 'pseudo_low': {'title': 'Pseudo-color
    image, pan sharpened (Uses IR and Visual bands, 542 mapping)'},
    'visual_low': {'title': 'Real-color image, pan sharpened (Uses the visual
    bands, 321 mapping)'}, 'visual_bright': {'title': 'Real-color image (Uses
    the visual bands, 321 mapping), gamma 1.5'}}

Available methods, their URLs, and available formats::

    >>> [op.name for op in wms.operations]
    ['GetTileService', 'GetCapabilities', 'GetMap']
    >>> wms.getOperationByName('GetMap').methods
    {'Get': {'url': 'http://wms.jpl.nasa.gov/wms.cgi?'}}
    >>> wms.getOperationByName('GetMap').formatOptions
    ['image/jpeg', 'image/png', 'image/geotiff', 'image/tiff']

That's everything needed to make a request for imagery::

    >>> img = wms.getmap(   layers=['global_mosaic'],
    ...                     styles=['visual_bright'],
    ...                     srs='EPSG:4326',
    ...                     bbox=(-112, 36, -106, 41),
    ...                     size=(300, 250),
    ...                     format='image/jpeg',
    ...                     transparent=True
    ...                     )
    >>> out = open('jpl_mosaic_visb.jpg', 'wb')
    >>> out.write(img.read())
    >>> out.close()
```

A very similar API exists for WebFeatureService. See
tests/wfs_MapServerWFSCapabilities.txt for details.

There is also support for Web Coverage Service (WCS), Catalogue
Service for the Web (CSW), Web Processing Service (WPS), and Web
Map Tile Service (WMTS). Some of those are beta quality.

Logging
-------
OWSLib logs messages to module-based named Python loggers. You may
configure your application to use the log messages like so:

```python
    >>> import logging
    >>> LOGGER = logging.getLogger(__name__)
    >>> # Add formatting and handlers as needed, for example to log to the console
    >>> ch = logging.StreamHandler()
    >>> ch.setLevel(logging.DEBUG)
    >>> ch.setFormatter(logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s'))
    >>> # add the handler to the logger
    >>> LOGGER.addHandler(ch)
    >>> LOGGER.setLevel(logging.DEBUG)
```

Releasing
---------

```bash
  # update version
  vi owslib/__init__.py
  git commit -m 'update release version' owslib/__init__.py
  # push changes
  git push origin master
  git tag -a x.y.z -m 'tagging OWSLib release x.y.z'
  # push tag
  git push --tags
  # update on PyPI (must be a maintainer)
  rm -fr build dist *.egg-info
  python3 setup.py sdist bdist_wheel --universal
  twine upload dist/*
```

Support
-------

- https://lists.osgeo.org/mailman/listinfo/owslib-users
- https://lists.osgeo.org/mailman/listinfo/owslib-devel


=========================
Spectral Parameterization
=========================

|ProjectStatus| |Version| |BuildStatus| |Coverage| |License| |PythonVersions| |Publication|

.. |ProjectStatus| image:: http://www.repostatus.org/badges/latest/active.svg
   :target: https://www.repostatus.org/#active
   :alt: project status

.. |Version| image:: https://img.shields.io/pypi/v/fooof.svg
   :target: https://pypi.python.org/pypi/fooof/
   :alt: version

.. |BuildStatus| image:: https://github.com/fooof-tools/fooof/actions/workflows/build.yml/badge.svg
   :target: https://github.com/fooof-tools/fooof/actions/workflows/build.yml
   :alt: build status

.. |Coverage| image:: https://codecov.io/gh/fooof-tools/fooof/branch/main/graph/badge.svg
   :target: https://codecov.io/gh/fooof-tools/fooof
   :alt: coverage

.. |License| image:: https://img.shields.io/pypi/l/fooof.svg
   :target: https://opensource.org/licenses/Apache-2.0
   :alt: license

.. |PythonVersions| image:: https://img.shields.io/pypi/pyversions/fooof.svg
   :target: https://pypi.python.org/pypi/fooof/
   :alt: python versions

.. |Publication| image:: https://img.shields.io/badge/paper-nn10.1038-informational.svg
   :target: https://doi.org/10.1038/s41593-020-00744-x
   :alt: publication

Spectral parameterization (`specparam`, formerly `fooof`) is a fast, efficient, and physiologically-informed tool to parameterize neural power spectra.

WARNING: this Github repository has been updated to a major update / breaking change from previous releases, which were under the `fooof` name, and now contains major breaking update for the new `specparam` version of the code. The new version is not fully released, though a test version is available (see installation instructions below).

Overview
--------

The power spectrum model conceives of a model of the power spectrum as a combination of two distinct functional processes:

- An aperiodic component, reflecting 1/f like characteristics, with
- A variable number of periodic components (putative oscillations), as peaks rising above the aperiodic component

This model driven approach can be used to measure periodic and aperiodic properties of electrophysiological data,
including EEG, MEG, ECoG and LFP data.

The benefit of fitting a model in order to measure putative oscillations, is that peaks in the power spectrum are
characterized in terms of their specific center frequency, power and bandwidth without requiring predefining
specific bands of interest and controlling for the aperiodic component.
The model also returns a measure of this aperiodic components of the signal, allowing for measuring and
comparison of 1/f-like components of the signal within and between subjects.

specparam (upcoming version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We are currently in the process of a major update to this tool, that includes a name changes (fooof -> specparam), and full rewrite of the code. This means that the new version will be incompatible with prior versions (in terms of the code having different names, and previous code no longer running as written), though note that the exact same procedures will be available (spectra can be fit in a way expected to give the same results), as well many new features.

The new version is called `specparam` (spectral parameterization). There is a release candidate available for testing (see installation instructions).

fooof (stable version)
~~~~~~~~~~~~~~~~~~~~~~

The fooof naming scheme, with most recent stable version 1.1 is the current main release, and is fully functional and stable, including everything that was introduced under the fooof name.

Which version should I use?
~~~~~~~~~~~~~~~~~~~~~~~~~~~

The previous release version, fooof, is fully functional, and projects that are already using it might as well stick with that, unless any of the new functionality in specparam is particularly important. For projects that are just starting, the new specparam version may be of interest if some of the new features are of interest (e.g. time-resolved estimations), though note that as release candidates, the release are not guaranteed to be stable (future updates may make breaking changes). Note that for the same model and settings, fooof and specparam should be exactly equivalent, so in terms of outputs there should be no difference in choosing one or the other.

Documentation
-------------

The `specparam` package includes a full set of code documentation.

specparam (upcoming version)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To see the documentation for the candidate 2.0 release, see
`here <https://specparam-tools.github.io/>`_.

fooof (stable version)
~~~~~~~~~~~~~~~~~~~~~~

Documentation is available on the
`documentation site <https://fooof-tools.github.io/>`_.

This documentation includes:

- `Motivations <https://fooof-tools.github.io/fooof/auto_motivations/index.html>`_:
  with motivating examples of why we recommend parameterizing neural power spectra
- `Tutorials <https://fooof-tools.github.io/fooof/auto_tutorials/index.html>`_:
  with a step-by-step guide through the model and how to apply it
- `Examples <https://fooof-tools.github.io/fooof/auto_examples/index.html>`_:
  demonstrating example analyses and use cases, and other functionality
- `API list <https://fooof-tools.github.io/fooof/api.html>`_:
  which lists and describes all the code and functionality available in the module
- `FAQ <https://fooof-tools.github.io/fooof/faq.html>`_:
  answering frequency asked questions
- `Glossary <https://fooof-tools.github.io/fooof/glossary.html>`_:
  which defines all the key terms used in the module
- `Reference <https://fooof-tools.github.io/fooof/reference.html>`_:
  with information for how to reference and report on using the module

Dependencies
------------

`specparam` is written in Python, and requires Python >= 3.7 to run.

It has the following required dependencies:

- `numpy <https://github.com/numpy/numpy>`_
- `scipy <https://github.com/scipy/scipy>`_ >= 0.19

There are also optional dependencies, which are not required for model fitting itself, but offer extra functionality:

- `matplotlib <https://github.com/matplotlib/matplotlib>`_ is needed to visualize data and model fits
- `tqdm <https://github.com/tqdm/tqdm>`_ is needed to print progress bars when fitting many models
- `pandas <https://github.com/pandas-dev/pandas>`_ is needed for exporting model fit results to dataframes
- `pytest <https://github.com/pytest-dev/pytest>`_ is needed to run the test suite locally

We recommend using the `Anaconda <https://www.anaconda.com/distribution/>`_ distribution to manage these requirements.

Installation
------------

specparam / fooof can be installed using pip.

specparam (test version)
~~~~~~~~~~~~~~~~~~~~~~~~

To install the current release candidate version for the new 2.0 version, you can do:

.. code-block:: shell

    $ pip install specparam

The above will install the most recent release candidate.

NOTE: specparam is currently available as a 'release candidate', meaning it is not finalized and fully released yet.
This means it may not yet have all features that the ultimate 2.0 version will include, and things are not strictly
guaranteed to stay the same (there may be further breaking changes in the ultimate 2.0 release).

fooof (stable version)
~~~~~~~~~~~~~~~~~~~~~~

The current major release is the 1.X.X series, which is a breaking change from the prior 0.X.X series.

Check the `changelog <https://fooof-tools.github.io/fooof/changelog.html>`_ for notes on updating to the new version.

**Stable Version**

To install the latest stable release, use pip:

.. code-block:: shell

    $ pip install fooof

The module can also be installed with conda, from the conda-forge channel:

.. code-block:: shell

    $ conda install -c conda-forge fooof

**Development Version**

To get the current development version, first clone this repository:

.. code-block:: shell

    $ git clone https://github.com/fooof-tools/fooof

To install this cloned copy, move into the directory you just cloned, and run:

.. code-block:: shell

    $ pip install .

**Editable Version**

To install an editable version, download the development version as above, and run:

.. code-block:: shell

    $ pip install -e .

Other Language Support
----------------------

The original implementation of `specparam`, available in this repository, is implemented in Python.

If you wish to run specparam from another language, there are a couple potential options:

- a `wrapper`, which allows for running the Python code from another language
- a `reimplementation`, which reflects a new implementation of the specparam algorithm in another language

Below are listed some examples of wrappers and/or re-implementations in other languages (non-exhaustive).

Matlab
~~~~~~

In Matlab, there is a reimplementation available in common toolboxes:
- The `Brainstorm <https://neuroimage.usc.edu/brainstorm/Introduction>`_ toolbox has a reimplementation of specparam (see the `Brainstorm fooof tutorial <https://neuroimage.usc.edu/brainstorm/Tutorials/Fooof>`_)
- The `Fieldtrip <https://www.fieldtriptoolbox.org/>`_ toolbox also uses the same reimplementation (see the `Fieldtrip fooof tutorial <https://www.fieldtriptoolbox.org/example/fooof/>`_)

There is also a Matlab wrapper in the `fooof_mat <http://github.com/fooof-tools/fooof_mat>`_ repository.

Note that another option is to use Python FOOOF within a Matlab pipeline, as explored in the
`mat_py_mat <https://github.com/fooof-tools/mat_py_mat>`_ repository.

Other Languages
~~~~~~~~~~~~~~~

Other languages with wrappers include:

- Julia, for which there is a `fooof wrapper <https://juliahub.com/ui/Packages/PyFOOOF/Ng8hN/0.1.0>`_
- R, in which fooof can be run using `reticulate <https://rstudio.github.io/reticulate/>`_, as `shown here <https://github.com/fooof-tools/DevelopmentalDemo/tree/main/R>`_

Reference
---------

If you use this code in your project, please cite::

    Donoghue T, Haller M, Peterson EJ, Varma P, Sebastian P, Gao R, Noto T, Lara AH, Wallis JD,
    Knight RT, Shestyuk A, & Voytek B (2020). Parameterizing neural power spectra into periodic
    and aperiodic components. Nature Neuroscience, 23, 1655-1665.
    DOI: 10.1038/s41593-020-00744-x

Direct link: https://doi.org/10.1038/s41593-020-00744-x

More information for how to cite this method can be found on the
`reference page <https://fooof-tools.github.io/fooof/reference.html>`_.

Code and analyses from the paper are also available in the
`paper repository <https://github.com/fooof-tools/Paper>`_.

Contribute
----------

This project welcomes and encourages contributions from the community!

To file bug reports and/or ask questions about this project, please use the
`Github issue tracker <https://github.com/fooof-tools/fooof/issues>`_.

To see and get involved in discussions about the module, check out:

- the `issues board <https://github.com/fooof-tools/fooof/issues>`_ for topics relating to code updates, bugs, and fixes
- the `development page <https://github.com/fooof-tools/Development>`_ for discussion of potential major updates to the module

When interacting with this project, please use the
`contribution guidelines <https://github.com/fooof-tools/fooof/blob/main/CONTRIBUTING.md>`_
and follow the
`code of conduct <https://github.com/fooof-tools/fooof/blob/main/CODE_OF_CONDUCT.md>`_.

Quickstart
----------

This module is object oriented, and uses a similar approach as used in scikit-learn.

The algorithm works on frequency representations, that is power spectra in linear space.

**Fitting a Single Power Spectrum**

With a power spectrum loaded (with 'freqs' storing frequency values, and 'spectrum' storing
the power spectrum, both as 1D arrays in linear space) parameterization can be done as follows:

.. code-block:: python

    # Import the model object
    from specparam import SpectralModel

    # Initialize model object
    fm = SpectralModel()

    # Define frequency range across which to model the spectrum
    freq_range = [3, 40]

    # Parameterize the power spectrum, and print out a report
    fm.report(freqs, spectrum, freq_range)

SpectralModel.report() fits the model, plots the original power spectrum with the associated model fit,
and prints out the parameters of the model fit for both the aperiodic component, and parameters for
any identified peaks, reflecting periodic components.

Example output for the report of a parameterized fit on an individual power spectrum:

.. image:: https://raw.githubusercontent.com/fooof-tools/fooof/main/doc/img/FOOOF_report.png

**Defining the model Settings**

The settings for the algorithm are:

* ``peak_width_limits`` sets the possible lower- and upper-bounds for the fitted peak widths.
* ``max_n_peaks`` sets the maximum number of peaks to fit.
* ``min_peak_height`` sets an absolute limit on the minimum height (above aperiodic) for any extracted peak.
* ``peak_threshold`` sets a relative threshold above which a peak height must cross to be included in the model.
* ``aperiodic_mode`` defines the approach to use to parameterize the aperiodic component.

These settings can be defined when initializing the model, for example:

.. code-block:: python

    # Initialize a model object with defined settings
    fm = SpectralModel(peak_width_limits=[1.0, 8.0], max_n_peaks=6, min_peak_height=0.1,
                       peak_threshold=2.0, aperiodic_mode='fixed')

**Fitting a Group of Power Spectra**

Next is an example workflow for fitting a group of neural power spectra.
In this case, 'freqs' is again a 1D array of frequency values, and 'spectra' is a 2D array of power spectra.
We can fit the group of power spectra by doing:

.. code-block:: python

    # Initialize a SpectralGroupModel object, specifying some parameters
    fg = SpectralGroupModel(peak_width_limits=[1.0, 8.0], max_n_peaks=8)

    # Fit models across the matrix of power spectra
    fg.fit(freqs, spectra)

    # Create and save out a report summarizing the results across the group of power spectra
    fg.save_report()

    # Save out results for further analysis later
    fg.save(file_name='group_results', save_results=True)

Example output from using SpectralGroupModel across a group of power spectra:

.. image:: https://raw.githubusercontent.com/fooof-tools/fooof/main/doc/img/FOOOFGroup_report.png

**Other Functionality**

The module also includes functionality for fitting the model to matrices of multiple power spectra,
saving and loading results, creating reports describing model fits, analyzing model outputs,
plotting models and parameters, and simulating power spectra, all of which is described in the
`documentation <https://fooof-tools.github.io/fooof/>`_.

Funding
-------

Supported by NIH award R01 GM134363 from the
`NIGMS <https://www.nigms.nih.gov/>`_.

.. image:: https://www.nih.gov/sites/all/themes/nih/images/nih-logo-color.png
  :width: 400

|


# Paddle2ONNX

简体中文 | [English](README_en.md)

# 1 Paddle2ONNX 简介

Paddle2ONNX 支持将 **PaddlePaddle** 模型格式转化到 **ONNX** 模型格式。通过 ONNX 可以完成将 Paddle 模型到多种推理引擎的部署，包括 TensorRT/OpenVINO/MNN/TNN/NCNN，以及其它对 ONNX 开源格式进行支持的推理引擎或硬件。

# 2 Paddle2ONNX 环境依赖

Paddle2ONNX 本身不依赖其他组件，但是我们建议您在以下环境下使用 Paddle2ONNX ：

- PaddlePaddle == 2.6.0
- onnxruntime >= 1.10.0

# 3 安装 Paddle2ONNX

如果您只是想要安装 Paddle2ONNX 且没有二次开发的需求，你可以通过执行以下代码来快速安装 Paddle2ONNX

```
pip install paddle2onnx
```

如果你希望对 Paddle2ONNX 进行二次开发，请按照[Github 源码安装方式](docs/zh/compile_local.md)编译Paddle2ONNX。

# 4 快速使用教程

## 4.1 获取PaddlePaddle部署模型

Paddle2ONNX 在导出模型时，需要传入部署模型格式，包括两个文件

- `model_name.pdmodel`: 表示模型结构
- `model_name.pdiparams`: 表示模型参数

## 4.2 调整Paddle模型

如果对Paddle模型的输入输出需要做调整，可以前往[Paddle 相关工具](./tools/paddle/README.md)查看教程。

## 4.3 使用命令行转换 PaddlePaddle 模型

你可以通过使用命令行并通过以下命令将Paddle模型转换为ONNX模型

```bash
paddle2onnx --model_dir saved_inference_model \
            --model_filename model.pdmodel \
            --params_filename model.pdiparams \
            --save_file model.onnx
```

可调整的转换参数如下表:

| 参数                         | 参数说明                                                                                |
|----------------------------|-----------------------------------------------------------------------------------------------------------------|
| --model_dir                | 配置包含 Paddle 模型的目录路径                                                                                |
| --model_filename           | **[可选]** 配置位于 `--model_dir` 下存储网络结构的文件名                                                                         |
| --params_filename          | **[可选]** 配置位于 `--model_dir` 下存储模型参数的文件名称                                                                        |
| --save_file                | 指定转换后的模型保存目录路径                                                                                |
| --opset_version            | **[可选]** 配置转换为 ONNX 的 OpSet 版本，目前支持 7~16 等多个版本，默认为 9                                                            |
| --enable_onnx_checker      | **[可选]**  配置是否检查导出为 ONNX 模型的正确性, 建议打开此开关， 默认为 False                                                             |
| --enable_auto_update_opset | **[可选]**  是否开启 opset version 自动升级功能，当低版本 opset 无法转换时，自动选择更高版本的 opset进行转换， 默认为 True                              |
| --deploy_backend           | **[可选]** 量化模型部署的推理引擎，支持 onnxruntime、tensorrt 或 others，当选择 others 时，所有的量化信息存储于 max_range.txt 文件中，默认为 onnxruntime |
| --save_calibration_file    | **[可选]** TensorRT 8.X版本部署量化模型需要读取的 cache 文件的保存路径，默认为 calibration.cache                                          |
| --version                  | **[可选]** 查看 paddle2onnx 版本                                                                                |
| --external_filename        | **[可选]** 当导出的 ONNX 模型大于 2G 时，需要设置 external data 的存储路径，推荐设置为：external_data                                       |
| --export_fp16_model        | **[可选]** 是否将导出的 ONNX 的模型转换为 FP16 格式，并用 ONNXRuntime-GPU 加速推理，默认为 False                                           |
| --custom_ops               | **[可选]** 将 Paddle OP 导出为 ONNX 的 Custom OP，例如：--custom_ops '{"paddle_op":"onnx_op"}，默认为 {}                       |

## 4.4 裁剪ONNX

如果你需要调整 ONNX 模型，请参考 [ONNX 相关工具](./tools/onnx/README.md)

## 4.5 优化ONNX

如你对导出的 ONNX 模型有优化的需求，推荐使用 `onnx-simplifier`，也可使用如下命令对模型进行优化

```
python -m paddle2onnx.optimize --input_model model.onnx --output_model new_model.onnx
```

# 5 代码贡献

繁荣的生态需要大家的携手共建，开发者可以参考 [Paddle2ONNX 贡献指南](./docs/zh/Paddle2ONNX_Development_Guide.md) 来为 Paddle2ONNX 贡献代码。

# 6 License

Provided under the [Apache-2.0 license](https://github.com/PaddlePaddle/paddle-onnx/blob/develop/LICENSE).

# 7 感谢捐赠

* 感谢 PaddlePaddle 团队提供服务器支持 Paddle2ONNX 的 CI 建设。
* 感谢社区用户 [chenwhql](https://github.com/chenwhql), [luotao1](https://github.com/luotao1), [goocody](https://github.com/goocody), [jeff41404](https://github.com/jeff41404), [jzhang553](https://github.com/jzhang533), [ZhengBicheng](https://github.com/ZhengBicheng) 于2024年03月28日向 Paddle2ONNX PMC 捐赠共 10000 元人名币用于 Paddle2ONNX 的发展。