Python: cannot read json-file with tinydb

Question

I've got the following json-file:

tag.tg

[{"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A0;          \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
{"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A0;           \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
{"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A0;            \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
{"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A1;          \\/**< The derived gain, A1 = -Kp - 2Kd. *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
{"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A1;$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
{"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A1;            \\/**< The derived gain, A1 = -Kp - 2Kd. *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
{"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A1;           \\/**< The derived gain A1 = -Kp - 2Kd | Kd.*\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
{"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A2;          \\/**< The derived gain, A2 = Kd . *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
{"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A2;$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
{"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A2;            \\/**< The derived gain, A2 = Kd . *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
{"_type": "tag", "name": "ABFSR", "path": "Drivers\\CMSIS\\Include\\core_cm7.h", "pattern": "/^  __IOM uint32_t ABFSR;                  \\/*!< Offset: 0x2A8 (R\\/W)  Auxiliary Bus Fault Status /", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee350000a08", "scopeKind": "struct"},
{"_type": "tag", "name": "ABR", "path": "Drivers\\CMSIS\\Device\\ST\\STM32F7xx\\Include\\stm32f767xx.h", "pattern": "/^  __IO uint32_t ABR;      \\/*!< QUADSPI Alternate Bytes register,                   Address offs/", "typeref": "__IO uint32_t", "kind": "member", "scope": "__anonea3271f52708", "scopeKind": "struct"},
{"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm3.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee33efc1008", "scopeKind": "struct"},
{"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm4.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee3433d1008", "scopeKind": "struct"},
{"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm7.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee350001008", "scopeKind": "struct"},
{"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_sc300.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anonccff54a21008", "scopeKind": "struct"},
{"_type": "tag", "name": "ACR", "path": "Drivers\\CMSIS\\Device\\ST\\STM32F7xx\\Include\\stm32f767xx.h", "pattern": "/^  __IO uint32_t ACR;      \\/*!< FLASH access control register,     Address offset: 0x00 *\\/$/", "typeref": "__IO uint32_t", "kind": "member", "scope": "__anonea3271f51408", "scopeKind": "struct"}
]

I've checked the validity of the json-file on https://jsonlint.com/, and it looks okay. Right now, I'm trying to load the json-file into a TinyDB database in Python:

from tinydb import TinyDB, Query

db = TinyDB("tag.tg")
Line = Query()
result = db.search(Line.name == "ADC_CCR_DELAY_2")
print(result)

The code fails at the second line db = TinyDB("tag.tg"). I get the following error:

TypeError: list indices must be integers or slices, not str

What did I do wrong? I'm exactly following the tinydb docs https://pypi.python.org/pypi/tinydb:

>>> from tinydb import TinyDB, Query
>>> db = TinyDB('/path/to/db.json')

---

Note:
I'm using Python 3.6.1

Answer 1

TinyDB expects an indexed table document, not a list. Unless you want to write a custom middleware for your TinyDB, you'll either have to modify your JSON to:

{
  "_default":{
    "1": {"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A0;          \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
    "2": {"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A0;           \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
    "3": {"_type": "tag", "name": "A0", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A0;            \\/**< The derived gain, A0 = Kp + Ki + Kd . *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
    "4": {"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A1;          \\/**< The derived gain, A1 = -Kp - 2Kd. *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
    "5": {"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A1;$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
    "6": {"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A1;            \\/**< The derived gain, A1 = -Kp - 2Kd. *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
    "7": {"_type": "tag", "name": "A1", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A1;           \\/**< The derived gain A1 = -Kp - 2Kd | Kd.*\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
    "8": {"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    float32_t A2;          \\/**< The derived gain, A2 = Kd . *\\/$/", "typeref": "float32_t", "kind": "member", "scope": "__anond9a59aba0f08", "scopeKind": "struct"},
    "9": {"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q15_t A2;$/", "typeref": "q15_t", "kind": "member", "scope": "__anond9a59aba0d08", "scopeKind": "struct"},
    "10": {"_type": "tag", "name": "A2", "path": "Drivers\\CMSIS\\Include\\arm_math.h", "pattern": "/^    q31_t A2;            \\/**< The derived gain, A2 = Kd . *\\/$/", "typeref": "q31_t", "kind": "member", "scope": "__anond9a59aba0e08", "scopeKind": "struct"},
    "11": {"_type": "tag", "name": "ABFSR", "path": "Drivers\\CMSIS\\Include\\core_cm7.h", "pattern": "/^  __IOM uint32_t ABFSR;                  \\/*!< Offset: 0x2A8 (R\\/W)  Auxiliary Bus Fault Status /", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee350000a08", "scopeKind": "struct"},
    "12": {"_type": "tag", "name": "ABR", "path": "Drivers\\CMSIS\\Device\\ST\\STM32F7xx\\Include\\stm32f767xx.h", "pattern": "/^  __IO uint32_t ABR;      \\/*!< QUADSPI Alternate Bytes register,                   Address offs/", "typeref": "__IO uint32_t", "kind": "member", "scope": "__anonea3271f52708", "scopeKind": "struct"},
    "13": {"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm3.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee33efc1008", "scopeKind": "struct"},
    "14": {"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm4.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee3433d1008", "scopeKind": "struct"},
    "15": {"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_cm7.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anoncee350001008", "scopeKind": "struct"},
    "16": {"_type": "tag", "name": "ACPR", "path": "Drivers\\CMSIS\\Include\\core_sc300.h", "pattern": "/^  __IOM uint32_t ACPR;                   \\/*!< Offset: 0x010 (R\\/W)  Asynchronous Clock Prescale/", "typeref": "__IOM uint32_t", "kind": "member", "scope": "__anonccff54a21008", "scopeKind": "struct"},
    "17": {"_type": "tag", "name": "ACR", "path": "Drivers\\CMSIS\\Device\\ST\\STM32F7xx\\Include\\stm32f767xx.h", "pattern": "/^  __IO uint32_t ACR;      \\/*!< FLASH access control register,     Address offset: 0x00 *\\/$/", "typeref": "__IO uint32_t", "kind": "member", "scope": "__anonea3271f51408", "scopeKind": "struct"}
  }
}

Or write a custom 'importer' for your existing data:

import json
import tinydb

db = tinydb.TinyDB("db_storage.json")  # create a new storage for the database

with open("tag.tg", "r") as f:  # open the unmodified `tag.tg`
    json_data = json.load(f)  # parse its JSON

for entry in json_data:  # iterate over each entry in the `tag.tg`
    db.insert(entry)  # insert it in the DB

UPDATE: Some additional info, although going a bit outside of this question's scope.

As I've suggested in the comments, if you intend to use a huge amount of records/data, it might be a better idea to use a more optimized DB. If you need something self-encapsulating end delivered with Python itself, SQLite is not a bad option but remember that it's an SQL database and not a document store so you need to build your schema first, so based on your data:

import sqlite3

connection = sqlite3.connect("db_store.sqlite")  # 'connect' to the db_store.sqlite

cursor = connection.cursor()  # get a cursor
cursor.execute("""
    CREATE TABLE IF NOT EXISTS main
    (_type TEXT, name TEXT, path TEXT, pattern TEXT,
    typeref TEXT, kind TEXT, scope TEXT, scopeKind TEXT)
""")  # create `main` table if it doesn't exist

Then you can use a similar approach as with TinyDB to insert your tag.tg JSON - load it, parse it, insert its items:

import json

# load and parse the `tag.tg`
with open("tag.tg", "r") as f:  # open the unmodified `tag.tg`
    json_data = json.load(f)  # parse its JSON

# fields that we're interested in, you can omit some if you don't want them in the DB
fields = ("_type", "name", "path", "pattern", "typeref", "kind", "scope", "scopeKind")
# prepare the insert query
query = "INSERT INTO main ({}) VALUES ({})".format(",".join(fields),
                                                   ",".join(("?",) * len(fields)))
# now let's add the records to the SQLite table:
cursor = connection.cursor()  # get a cursor
cursor.executemany(query, ([e.get(field, None) for field in fields] for e in json_data))

And now you can do things like:

cursor = connection.cursor()  # get a cursor
cursor.execute("SELECT path FROM main WHERE name = 'ABFSR'")  # a generic SQL SELECT
path, = cursor.fetchone()  # get the first result
print("Path: " + path)  # Path: Drivers\CMSIS\Include\core_cm7.h

It will certainly be able to handle far more records than TinyDB, the only problematic thing might be importing your initial JSON data - if your system just can't handle loading and parsing the whole JSON structure at once you might want to look out for streaming JSON parsers, although if all of your data is structured as you've shown (a different JSON object on each line) one might take a swing at it by removing the surrounding [ and ] and comma separators and read & parse the file line by line.