TDDA: Test-Driven Data Analysis¶
TDDA uses file inputs (such as NumPy arrays or Pandas DataFrames) and a set of constraints that are stored as a JSON file.
Reference Tests supports the creation of regression tests based on either unittest or pytest.
Constraints is used to retrieve constraints from a (pandas) DataFrame, write them out as JSON and check whether records satisfy the constraints in the constraints file. It also supports tables in a variety of relational databases.
Rexpy is a tool for automatically deriving regular expressions from a column in a pandas DataFrame or from a (Python) list of examples. TDDA can also be used via the command line and is therefore accessible to anyone who works with data – regardless of whether they use Python, R, Excel, SQL or other languages and tools.
1. Imports¶
[1]:
from pathlib import Path
import pandas as pd
from tdda.constraints import detect_df, discover_df, verify_df
[2]:
df = pd.read_csv(
"https://raw.githubusercontent.com/kjam/data-cleaning-101/master/data/iot_example.csv",
)
2. Check data¶
With pandas.DataFrame.sample we display ten random data sets:
[3]:
df.sample(10)
[3]:
| timestamp | username | temperature | heartrate | build | latest | note | |
|---|---|---|---|---|---|---|---|
| 81591 | 2017-02-03T03:11:13 | briansmith | 29 | 69 | 76ca8eb4-33ff-3cb3-f710-58a9c06ec975 | 0 | NaN |
| 62704 | 2017-01-26T13:39:28 | taralee | 17 | 60 | d2e72090-7a02-1e44-09b6-7637ada562cd | 0 | update |
| 131624 | 2017-02-23T02:09:41 | carlsonjeremiah | 11 | 60 | 9b40fb9b-316c-b1b6-1d1e-b3e8c3fa7fad | 0 | user |
| 109988 | 2017-02-14T10:56:40 | justinmendez | 18 | 78 | d89af579-d2fd-eb5b-8ad2-12c53a62cb85 | 0 | update |
| 109040 | 2017-02-14T01:58:48 | nnorris | 13 | 61 | 5a00cab8-4fd3-d370-5e8b-c10147862dbe | 0 | NaN |
| 69304 | 2017-01-29T04:59:02 | sjohnson | 29 | 75 | b3479e08-07b9-d8d9-5b90-7b76bfb75acd | 1 | test |
| 139179 | 2017-02-26T02:41:40 | dale00 | 23 | 86 | 1e4ad2aa-fc45-5f2b-fdaa-b5552c250b21 | 1 | NaN |
| 50069 | 2017-01-21T12:15:46 | fostermichael | 16 | 68 | a682ccce-42b8-badf-b3d7-73180e766746 | 0 | NaN |
| 46103 | 2017-01-19T22:18:21 | fosterryan | 8 | 81 | add3f2fa-6039-3263-f526-a250ace1ecab | 1 | wake |
| 15108 | 2017-01-07T12:57:16 | stevenrobinson | 15 | 68 | e44fff6d-8390-9ecf-e952-2126e9fbe2ec | 1 | interval |
And with pandas.DataFrame.dtypes we display the data types for the individual columns:
[4]:
df.dtypes
[4]:
timestamp object
username object
temperature int64
heartrate int64
build object
latest int64
note object
dtype: object
3. Creating a constraints object¶
With discover_df a constraints object can be created.
[5]:
constraints = discover_df(df)
{
"creation_metadata": {
"local_time": "2026-09-18T13:15:32",
"utc_time": "2026-09-18T11:15:32+00:00",
"creator": "TDDA 3.0.01",
"host": "fay.local",
"user": "veit",
"n_records": 146397,
"n_selected": 146397
},
"fields": {
"timestamp": {
"type": "string",
"min_length": 19,
"max_length": 19,
"max_nulls": 0,
"no_duplicates": true
},
"username": {
"type": "string",
"min_length": 3,
"max_length": 21,
"max_nulls": 0
},
"temperature": {
"type": "int",
"min": 5,
"max": 29,
"sign": "positive",
"max_nulls": 0
},
"heartrate": {
"type": "int",
"min": 60,
"max": 89,
"sign": "positive",
"max_nulls": 0
},
"build": {
"type": "string",
"min_length": 36,
"max_length": 36,
"max_nulls": 0,
"no_duplicates": true
},
"latest": {
"type": "int",
"min": 0,
"max": 1,
"sign": "non-negative",
"max_nulls": 0
},
"note": {
"type": "string",
"min_length": 4,
"max_length": 8,
"allowed_values": [
"interval",
"sleep",
"test",
"update",
"user",
"wake"
]
}
},
"dataset": {
"allowed_fields": [],
"required_fields": [
"*"
]
}
}
[6]:
constraints
[6]:
<tdda.constraints.base.DatasetConstraints at 0x15d1b4440>
[7]:
constraints.fields
[7]:
Fields([('timestamp', <tdda.constraints.base.FieldConstraints at 0x12189bcb0>),
('username', <tdda.constraints.base.FieldConstraints at 0x121a05090>),
('temperature',
<tdda.constraints.base.FieldConstraints at 0x121a05450>),
('heartrate', <tdda.constraints.base.FieldConstraints at 0x1218775c0>),
('build', <tdda.constraints.base.FieldConstraints at 0x121877950>),
('latest', <tdda.constraints.base.FieldConstraints at 0x15d19d130>),
('note', <tdda.constraints.base.FieldConstraints at 0x121893350>)])
4. Writing the constraints into a file¶
[8]:
with Path.open("../../../data/iot_example.json", "w") as f:
f.write(constraints.to_json())
If we take a closer look at the file, we can see that, for example, a string with 19 characters is expected for the timestamp column and temperature expects integers with values from 5-29.
[9]:
!cat ../../../data/iot_example.json
{
"creation_metadata": {
"local_time": "2026-09-18T13:15:32",
"utc_time": "2026-09-18T11:15:32+00:00",
"creator": "TDDA 3.0.01",
"host": "fay.local",
"user": "veit",
"n_records": 146397,
"n_selected": 146397
},
"fields": {
"timestamp": {
"type": "string",
"min_length": 19,
"max_length": 19,
"max_nulls": 0,
"no_duplicates": true
},
"username": {
"type": "string",
"min_length": 3,
"max_length": 21,
"max_nulls": 0
},
"temperature": {
"type": "int",
"min": 5,
"max": 29,
"sign": "positive",
"max_nulls": 0
},
"heartrate": {
"type": "int",
"min": 60,
"max": 89,
"sign": "positive",
"max_nulls": 0
},
"build": {
"type": "string",
"min_length": 36,
"max_length": 36,
"max_nulls": 0,
"no_duplicates": true
},
"latest": {
"type": "int",
"min": 0,
"max": 1,
"sign": "non-negative",
"max_nulls": 0
},
"note": {
"type": "string",
"min_length": 4,
"max_length": 8,
"allowed_values": [
"interval",
"sleep",
"test",
"update",
"user",
"wake"
]
}
},
"dataset": {
"allowed_fields": [],
"required_fields": [
"*"
]
}
}
You can also use discover_df to try and identify regular expressions:
[10]:
constraints = discover_df(df, inc_rex=True)
{
"creation_metadata": {
"local_time": "2026-09-18T13:15:34",
"utc_time": "2026-09-18T11:15:34+00:00",
"creator": "TDDA 3.0.01",
"host": "fay.local",
"user": "veit",
"n_records": 146397,
"n_selected": 146397
},
"fields": {
"timestamp": {
"type": "string",
"min_length": 19,
"max_length": 19,
"max_nulls": 0,
"no_duplicates": true,
"rex": [
"^2017\\-([0-9]{2})\\-([0-9]{2})([A-Z])([0-9]{2}):([0-9]{2}):([0-9]{2})$"
]
},
"username": {
"type": "string",
"min_length": 3,
"max_length": 21,
"max_nulls": 0,
"rex": [
"^([a-z0-9]+)$"
]
},
"temperature": {
"type": "int",
"min": 5,
"max": 29,
"sign": "positive",
"max_nulls": 0
},
"heartrate": {
"type": "int",
"min": 60,
"max": 89,
"sign": "positive",
"max_nulls": 0
},
"build": {
"type": "string",
"min_length": 36,
"max_length": 36,
"max_nulls": 0,
"no_duplicates": true,
"rex": [
"^([0-9a-f]{8})\\-([0-9a-f]{4})\\-([0-9a-f]{4})\\-([0-9a-f]{4})\\-([0-9a-f]{12})$"
]
},
"latest": {
"type": "int",
"min": 0,
"max": 1,
"sign": "non-negative",
"max_nulls": 0
},
"note": {
"type": "string",
"min_length": 4,
"max_length": 8,
"allowed_values": [
"interval",
"sleep",
"test",
"update",
"user",
"wake"
],
"rex": [
"^([a-z]+)$"
]
}
},
"dataset": {
"allowed_fields": [],
"required_fields": [
"*"
]
}
}
[11]:
with Path.open("../../../data/iot_re_example.json", "w") as f:
f.write(constraints.to_json())
5. Checking data frames¶
To do this, we first read in a new csv file with pandas and then have ten data records output as examples:
[12]:
new_df = pd.read_csv(
"https://raw.githubusercontent.com/kjam/data-cleaning-101/master/data/iot_example_with_nulls.csv",
)
new_df.sample(10)
[12]:
| timestamp | username | temperature | heartrate | build | latest | note | |
|---|---|---|---|---|---|---|---|
| 3810 | 2017-01-03T00:49:23 | meganthomas | 6.0 | 80 | NaN | NaN | interval |
| 51938 | 2017-01-22T06:04:30 | hilldavid | 9.0 | 69 | d3d7c62d-77ab-c5cd-1553-53de7f60b8d4 | 1.0 | interval |
| 52942 | 2017-01-22T15:46:09 | ericfoster | 10.0 | 62 | 266573fa-68e4-fab9-565c-cdf3dfe92db0 | 1.0 | NaN |
| 28286 | 2017-01-12T19:11:20 | mendozaclayton | 24.0 | 78 | 5356f03b-c026-df0a-c52c-4c2097278da6 | 0.0 | interval |
| 139531 | 2017-02-26T06:03:09 | phillip19 | 11.0 | 80 | cfedd723-b381-6e17-8664-ca63f40afaf1 | 0.0 | sleep |
| 145529 | 2017-02-28T15:43:10 | michelletanner | NaN | 75 | 64c0a861-7746-2fb3-0ea4-46a5b56ba345 | 0.0 | sleep |
| 93860 | 2017-02-08T00:45:35 | darmstrong | 24.0 | 87 | fd76a017-97db-ed0c-f90b-80ed310eeae8 | 1.0 | NaN |
| 12898 | 2017-01-06T15:49:40 | allenjanet | 6.0 | 84 | adc008c2-c531-7e16-efd4-11f19c62f615 | 1.0 | sleep |
| 32239 | 2017-01-14T09:09:23 | barbaralambert | 10.0 | 64 | 0a82b250-871c-8655-1312-6ba2580d65d7 | 0.0 | update |
| 100230 | 2017-02-10T13:40:35 | jonathanbrown | 20.0 | 75 | cee5c7f3-27d3-5cbd-8b1a-f9be004b2932 | 1.0 | NaN |
We see several fields that are output as NaN. Now, to analyse this systematically, we apply verify_df to our new DataFrame. Here, passes returns the number of passed constraints, and failures returns the number of failed constraints.
[13]:
v = verify_df(new_df, "../../../data/iot_example.json")
[14]:
v
[14]:
<tdda.constraints.pd.constraints.PandasVerification at 0x15d1b5fd0>
[15]:
v.passes
[15]:
32
[16]:
v.failures
[16]:
3
We can also display which constraints passed and failed in which columns:
[17]:
print(str(v))
FIELDS:
timestamp: 0 failures 5 passes type ✓ min_length ✓ max_length ✓ max_nulls ✓ no_duplicates ✓
username: 0 failures 4 passes type ✓ min_length ✓ max_length ✓ max_nulls ✓
temperature: 1 failure 4 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✗
heartrate: 0 failures 5 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✓
build: 1 failure 4 passes type ✓ min_length ✓ max_length ✓ max_nulls ✗ no_duplicates ✓
latest: 1 failure 4 passes type ✓ min ✓ max ✓ sign ✓ max_nulls ✗
note: 0 failures 4 passes type ✓ min_length ✓ max_length ✓ allowed_values ✓
DATASET:
Extra (disallowed) fields: None
Missing (required) fields: None
SUMMARY:
Constrained Fields: 7
Failing Fields: 3 (42.86%)
Constraints: 35
Failing Constraints: 3 (8.57%)
Extra (disallowed) fields: 0
Missing (required) fields: 0
Alternatively, we can also display these results in tabular form:
[18]:
v.to_frame()
[18]:
| field | failures | passes | type | min | min_length | max | max_length | sign | max_nulls | no_duplicates | allowed_values | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | timestamp | 0 | 5 | True | NaN | True | NaN | True | NaN | True | True | NaN |
| 1 | username | 0 | 4 | True | NaN | True | NaN | True | NaN | True | NaN | NaN |
| 2 | temperature | 1 | 4 | True | True | NaN | True | NaN | True | False | NaN | NaN |
| 3 | heartrate | 0 | 5 | True | True | NaN | True | NaN | True | True | NaN | NaN |
| 4 | build | 1 | 4 | True | NaN | True | NaN | True | NaN | False | True | NaN |
| 5 | latest | 1 | 4 | True | True | NaN | True | NaN | True | False | NaN | NaN |
| 6 | note | 0 | 4 | True | NaN | True | NaN | True | NaN | NaN | NaN | True |
6. Finding the faulty rows¶
tdda.constraints.pd.constraints.detect_df() detects records in the pandas DataFrame that violate one of the constraints in the provided JSON file. We can then call the detected() function on the created PandasDetection object to output the rows that are faulty:
[19]:
d = detect_df(new_df, "../../../data/iot_example.json")
d.detected()
[19]:
| n_failures | |
|---|---|
| Index | |
| 3 | 1 |
| 4 | 1 |
| 7 | 1 |
| 10 | 2 |
| 12 | 1 |
| ... | ... |
| 146385 | 1 |
| 146387 | 2 |
| 146391 | 2 |
| 146393 | 2 |
| 146394 | 1 |
77260 rows × 1 columns
We can display all incorrect data records by using only the part of the index from new_df that also appears in d.detected():
[20]:
d_index = d.detected().index
[21]:
new_df[new_df.index.isin(d_index)]
[21]:
| timestamp | username | temperature | heartrate | build | latest | note | |
|---|---|---|---|---|---|---|---|
| 3 | 2017-01-01T12:02:09 | eddierodriguez | 28.0 | 76 | NaN | 0.0 | update |
| 4 | 2017-01-01T12:02:36 | kenneth94 | 29.0 | 62 | 122f1c6a-403c-2221-6ed1-b5caa08f11e0 | NaN | NaN |
| 7 | 2017-01-01T12:04:35 | scott28 | 16.0 | 76 | 7a60219f-6621-e548-180e-ca69624f9824 | NaN | interval |
| 10 | 2017-01-01T12:06:21 | njohnson | NaN | 63 | e09b6001-125d-51cf-9c3f-9cb686c19d02 | NaN | NaN |
| 12 | 2017-01-01T12:07:41 | jessica48 | 22.0 | 83 | 03e1a07b-3e14-412c-3a69-6b45bc79f81c | NaN | update |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 146385 | 2017-02-28T23:53:59 | powelleric | 20.0 | 86 | 152eda10-676a-069c-b664-19443f2c8081 | NaN | test |
| 146387 | 2017-02-28T23:54:50 | jthompson | NaN | 66 | 8da10303-fe49-e313-8fda-0d5e79ded054 | NaN | update |
| 146391 | 2017-02-28T23:57:21 | aaronbecker | NaN | 87 | 7e52f4a8-345c-5ee0-e515-b8c392213062 | NaN | sleep |
| 146393 | 2017-02-28T23:58:43 | joelrusso | NaN | 89 | NaN | 0.0 | NaN |
| 146394 | 2017-02-28T23:59:23 | lellis | NaN | 84 | dac87426-e147-9c39-6e4c-790bb11f8fc9 | 0.0 | update |
77260 rows × 7 columns
[22]:
new_df[~new_df.index.isin(d_index)]
[22]:
| timestamp | username | temperature | heartrate | build | latest | note | |
|---|---|---|---|---|---|---|---|
| 0 | 2017-01-01T12:00:23 | michaelsmith | 12.0 | 67 | 4e6a7805-8faa-2768-6ef6-eb3198b483ac | 0.0 | interval |
| 1 | 2017-01-01T12:01:09 | kharrison | 6.0 | 78 | 7256b7b0-e502-f576-62ec-ed73533c9c84 | 0.0 | wake |
| 2 | 2017-01-01T12:01:34 | smithadam | 5.0 | 89 | 9226c94b-bb4b-a6c8-8e02-cb42b53e9c90 | 0.0 | NaN |
| 5 | 2017-01-01T12:03:04 | bryanttodd | 13.0 | 86 | 0897dbe5-9c5b-71ca-73a1-7586959ca198 | 0.0 | interval |
| 6 | 2017-01-01T12:03:51 | andrea98 | 17.0 | 81 | 1c07ab9b-5f66-137d-a74f-921a41001f4e | 1.0 | NaN |
| ... | ... | ... | ... | ... | ... | ... | ... |
| 146389 | 2017-02-28T23:56:05 | kathy63 | 5.0 | 88 | c2f76050-abd4-aee4-7bc0-3498325d0573 | 0.0 | NaN |
| 146390 | 2017-02-28T23:56:34 | cookallison | 16.0 | 84 | f0b0c1f9-900b-276c-bca9-ac4d4ec4e88e | 0.0 | user |
| 146392 | 2017-02-28T23:58:06 | mcontreras | 15.0 | 63 | 69e61a15-d2d0-47a7-1a27-e07b3eeeba10 | 0.0 | NaN |
| 146395 | 2017-02-28T23:59:48 | grayjasmin | 17.0 | 64 | 4911a589-3a15-4bbf-1de1-e5a69ab739da | 1.0 | update |
| 146396 | 2017-03-01T00:00:30 | jgreene | 23.0 | 70 | 4f95bbca-26a7-29e7-1f19-aaedf1a51741 | 0.0 | interval |
69137 rows × 7 columns