Meet Gor - Type: sqlog

Advent of SQL 2025 Day 15: Confirmation Phrase Dispatches

Sun, 28 Dec 2025 00:00:00 UTC

## Advent of SQL, Day 15 - Confirmation Phrase Dispatches We are on the final day of Advent of SQL! I can't believe it, I completed it! (with some help of course, I ~~don't~~ didn't know SQL very well. But this 15 days flipped it around.). I feel good and fresh to start the year to go deep in SQLite and databases! Let's solve the last day first, if it has somethings reamining to teach us! We need to make a few changes. SQLite doesn't have `->>` operator for extracting JSON values from the columns, so we need to use `json_extract` instead. So, replace this 9th line from and with: ```diff - marker_letter TEXT GENERATED ALWAYS AS (payload ->> 'marker') STORED, + marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, ``` Once we have it, we also need to remove the marker `::jsonb` from the each insert row, we can do that with `sed` or any other utility you like. ``` sed "s/'$.*$'::jsonb/'\1'/g" day15-inserts.sql > day15-inserts-sqlite.sql ``` ```sql DROP TABLE IF EXISTS incoming_dispatches; DROP TABLE IF EXISTS system_dispatches; CREATE TABLE system_dispatches ( id SERIAL PRIMARY KEY, system_id TEXT NOT NULL, dispatched_at TIMESTAMP NOT NULL, payload JSONB NOT NULL, marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, UNIQUE (system_id, dispatched_at, payload) ); CREATE TABLE incoming_dispatches ( system_id TEXT, dispatched_at TIMESTAMP, payload JSONB ); INSERT INTO system_dispatches (system_id, dispatched_at, payload) VALUES ('SYS-0081', '2025-12-21T06:02:26', '{"marker": "T", "source": "secondary"}'), ('SYS-0137', '2025-12-19T06:03:21', '{"marker": "D", "source": "secondary"}'), ('SYS-0237', '2025-12-19T06:23:37', '{"marker": "D", "source": "secondary"}'), ('SYS-0006', '2025-12-24T18:10:16', '{"marker": "K", "source": "secondary"}'), ('SYS-0170', '2025-12-19T06:17:24', '{"marker": "I", "source": "secondary"}'), ('SYS-0224', '2025-12-19T06:23:24', '{"marker": "S", "source": "secondary"}'), ('SYS-0007', '2025-12-24T18:10:06', '{"marker": "O", "source": "secondary"}'), ('SYS-0035', '2025-12-23T15:55:34', '{"marker": "C", "source": "secondary"}'), ('SYS-0010', '2025-12-23T15:55:09', '{"marker": "L", "source": "secondary"}'), ('SYS-0037', '2025-12-23T15:55:36', '{"marker": "D", "source": "secondary"}'), ('SYS-0001', '2025-12-24T08:02:00', '{"marker": "X", "source": "primary"}'), ('SYS-0225', '2025-12-19T06:23:25', '{"marker": "Q", "source": "secondary"}'), ('SYS-0095', '2025-12-21T06:02:40', '{"marker": "D", "source": "secondary"}'), ('SYS-0021', '2025-12-23T15:55:20', '{"marker": "W", "source": "secondary"}'), ('SYS-0001', '2025-12-24T08:10:00', '{"marker": "A", "source": "primary"}'), ('SYS-0009', '2025-12-24T08:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0119', '2025-12-19T06:03:03', '{"marker": "E", "source": "secondary"}'), ('SYS-0142', '2025-12-19T06:03:26', '{"marker": "O", "source": "secondary"}'), ('SYS-0048', '2025-12-23T15:55:30', '{"marker": "I", "source": "secondary"}'), ('SYS-0228', '2025-12-19T06:23:28', '{"marker": "G", "source": "secondary"}'), ('SYS-0092', '2025-12-21T06:02:37', '{"marker": "K", "source": "secondary"}'), ('SYS-0008', '2025-12-24T08:02:07', '{"marker": "B", "source": "primary"}'), ('SYS-0111', '2025-12-19T06:02:55', '{"marker": "W", "source": "secondary"}'), ('SYS-0106', '2025-12-19T06:02:50', '{"marker": "L", "source": "secondary"}'), ('SYS-0082', '2025-12-21T06:02:27', '{"marker": "O", "source": "secondary"}'), ('SYS-0180', '2025-12-19T06:17:31', '{"marker": "N", "source": "secondary"}'), ('SYS-0103', '2025-12-19T06:02:47', '{"marker": "F", "source": "secondary"}'), ('SYS-0096', '2025-12-21T06:02:41', '{"marker": "A", "source": "secondary"}'), ('SYS-0003', '2025-12-23T06:02:02', '{"marker": "Z", "source": "primary"}'), ('SYS-0157', '2025-12-19T06:17:11', '{"marker": "D", "source": "secondary"}'), ('SYS-0010', '2025-12-23T06:02:09', '{"marker": "C", "source": "primary"}'), ('SYS-0102', '2025-12-19T06:02:46', '{"marker": "O", "source": "secondary"}'), ('SYS-0072', '2025-12-23T06:02:18', '{"marker": "K", "source": "secondary"}'), ('SYS-0175', '2025-12-19T06:17:29', '{"marker": "D", "source": "secondary"}'), ('SYS-0214', '2025-12-19T06:23:14', '{"marker": "U", "source": "secondary"}'), ('SYS-0005', '2025-12-24T08:02:04', '{"marker": "P", "source": "primary"}'), ('SYS-0216', '2025-12-19T06:23:16', '{"marker": "A", "source": "secondary"}'), ('SYS-0047', '2025-12-23T15:55:46', '{"marker": "H", "source": "secondary"}'), ('SYS-0019', '2025-12-23T15:55:18', '{"marker": "H", "source": "secondary"}'), ('SYS-0043', '2025-12-23T15:55:42', '{"marker": "Q", "source": "secondary"}'), ('SYS-0039', '2025-12-23T15:55:38', '{"marker": "L", "source": "secondary"}'), ('SYS-0212', '2025-12-19T06:23:12', '{"marker": "K", "source": "secondary"}'), ('SYS-0133', '2025-12-19T06:03:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0005', '2025-12-24T08:10:04', '{"marker": "N", "source": "primary"}'), ('SYS-0011', '2025-12-24T08:02:10', '{"marker": "J", "source": "primary"}'), ('SYS-0080', '2025-12-21T06:02:25', '{"marker": "N", "source": "secondary"}'), ('SYS-0230', '2025-12-19T06:23:30', '{"marker": "I", "source": "secondary"}'), ('SYS-0100', '2025-12-21T06:02:45', '{"marker": "N", "source": "secondary"}'), ('SYS-0061', '2025-12-23T06:02:07', '{"marker": "T", "source": "secondary"}'), ('SYS-0022', '2025-12-23T15:55:21', '{"marker": "K", "source": "secondary"}'), ('SYS-0149', '2025-12-19T06:17:03', '{"marker": "H", "source": "secondary"}'), ('SYS-0009', '2025-12-23T15:55:08', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-24T08:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0132', '2025-12-19T06:03:16', '{"marker": "K", "source": "secondary"}'), ('SYS-0033', '2025-12-23T15:55:32', '{"marker": "W", "source": "secondary"}'), ('SYS-0193', '2025-12-19T06:17:44', '{"marker": "C", "source": "secondary"}'), ('SYS-0167', '2025-12-19T06:17:21', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-24T08:10:07', '{"marker": "F", "source": "primary"}'), ('SYS-0186', '2025-12-19T06:17:37', '{"marker": "L", "source": "secondary"}'), ('SYS-0078', '2025-12-21T06:02:23', '{"marker": "V", "source": "secondary"}'), ('SYS-0085', '2025-12-21T06:02:30', '{"marker": "Q", "source": "secondary"}'), ('SYS-0046', '2025-12-23T15:55:45', '{"marker": "G", "source": "secondary"}'), ('SYS-0054', '2025-12-23T15:55:36', '{"marker": "W", "source": "secondary"}'), ('SYS-0235', '2025-12-19T06:23:35', '{"marker": "D", "source": "secondary"}'), ('SYS-0089', '2025-12-21T06:02:34', '{"marker": "H", "source": "secondary"}'), ('SYS-0011', '2025-12-23T06:02:10', '{"marker": "J", "source": "primary"}'), ('SYS-0034', '2025-12-23T15:55:33', '{"marker": "K", "source": "secondary"}'), ('SYS-0129', '2025-12-19T06:03:13', '{"marker": "H", "source": "secondary"}'), ('SYS-0211', '2025-12-19T06:23:11', '{"marker": "W", "source": "secondary"}'), ('SYS-0151', '2025-12-19T06:17:05', '{"marker": "W", "source": "secondary"}'), ('SYS-0204', '2025-12-19T06:23:04', '{"marker": "S", "source": "secondary"}'), ('SYS-0213', '2025-12-19T06:23:13', '{"marker": "C", "source": "secondary"}'), ('SYS-0147', '2025-12-19T06:03:31', '{"marker": "A", "source": "secondary"}'), ('SYS-0140', '2025-12-19T06:03:24', '{"marker": "N", "source": "secondary"}'), ('SYS-0123', '2025-12-19T06:03:07', '{"marker": "F", "source": "secondary"}'), ('SYS-0052', '2025-12-23T15:55:34', '{"marker": "H", "source": "secondary"}'), ('SYS-0148', '2025-12-19T06:17:02', '{"marker": "G", "source": "secondary"}'), ('SYS-0059', '2025-12-23T06:02:05', '{"marker": "E", "source": "secondary"}'), ('SYS-0027', '2025-12-23T15:55:26', '{"marker": "K", "source": "secondary"}'), ('SYS-0055', '2025-12-23T15:55:37', '{"marker": "K", "source": "secondary"}'), ('SYS-0155', '2025-12-19T06:17:09', '{"marker": "D", "source": "secondary"}'), ('SYS-0107', '2025-12-19T06:02:51', '{"marker": "A", "source": "secondary"}'), ('SYS-0173', '2025-12-19T06:17:27', '{"marker": "C", "source": "secondary"}'), ('SYS-0007', '2025-12-23T06:02:06', '{"marker": "K", "source": "primary"}'), ('SYS-0016', '2025-12-23T15:55:15', '{"marker": "L", "source": "secondary"}'), ('SYS-0168', '2025-12-19T06:17:22', '{"marker": "G", "source": "secondary"}'), ('SYS-0196', '2025-12-19T06:17:47', '{"marker": "A", "source": "secondary"}'), ('SYS-0234', '2025-12-19T06:23:34', '{"marker": "U", "source": "secondary"}'), ('SYS-0210', '2025-12-19T06:23:10', '{"marker": "I", "source": "secondary"}'), ('SYS-0135', '2025-12-19T06:03:19', '{"marker": "D", "source": "secondary"}'), ('SYS-0150', '2025-12-19T06:17:04', '{"marker": "I", "source": "secondary"}'), ('SYS-0091', '2025-12-21T06:02:36', '{"marker": "W", "source": "secondary"}'), ('SYS-0154', '2025-12-19T06:17:08', '{"marker": "U", "source": "secondary"}'), ('SYS-0223', '2025-12-19T06:23:23', '{"marker": "F", "source": "secondary"}'), ('SYS-0020', '2025-12-23T15:55:19', '{"marker": "I", "source": "secondary"}'), ('SYS-0160', '2025-12-19T06:17:14', '{"marker": "N", "source": "secondary"}'), ('SYS-0128', '2025-12-19T06:03:12', '{"marker": "G", "source": "secondary"}'), ('SYS-0194', '2025-12-19T06:17:45', '{"marker": "U", "source": "secondary"}'), ('SYS-0231', '2025-12-19T06:23:31', '{"marker": "W", "source": "secondary"}'), ('SYS-0045', '2025-12-23T15:55:44', '{"marker": "A", "source": "secondary"}'), ('SYS-0190', '2025-12-19T06:17:41', '{"marker": "I", "source": "secondary"}'), ('SYS-0197', '2025-12-19T06:17:48', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-23T06:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0008', '2025-12-24T18:10:07', '{"marker": "F", "source": "secondary"}'), ('SYS-0158', '2025-12-19T06:17:12', '{"marker": "V", "source": "secondary"}'), ('SYS-0003', '2025-12-24T08:10:02', '{"marker": "V", "source": "primary"}'), ('SYS-0171', '2025-12-19T06:17:25', '{"marker": "W", "source": "secondary"}'), ('SYS-0229', '2025-12-19T06:23:29', '{"marker": "H", "source": "secondary"}'), ('SYS-0079', '2025-12-21T06:02:24', '{"marker": "E", "source": "secondary"}'), ('SYS-0215', '2025-12-19T06:23:15', '{"marker": "D", "source": "secondary"}'), ('SYS-0233', '2025-12-19T06:23:33', '{"marker": "C", "source": "secondary"}'), ('SYS-0060', '2025-12-23T06:02:06', '{"marker": "N", "source": "secondary"}'), ('SYS-0026', '2025-12-23T15:55:25', '{"marker": "W", "source": "secondary"}'), ('SYS-0125', '2025-12-19T06:03:09', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-24T18:10:00', '{"marker": "A", "source": "secondary"}'), ('SYS-0136', '2025-12-19T06:03:20', '{"marker": "A", "source": "secondary"}'), ('SYS-0010', '2025-12-24T08:02:09', '{"marker": "C", "source": "primary"}'), ('SYS-0010', '2025-12-24T08:10:09', '{"marker": "Q", "source": "primary"}'), ('SYS-0044', '2025-12-23T15:55:43', '{"marker": "L", "source": "secondary"}'), ('SYS-0115', '2025-12-19T06:02:59', '{"marker": "D", "source": "secondary"}'), ('SYS-0005', '2025-12-24T18:10:15', '{"marker": "W", "source": "secondary"}'), ('SYS-0094', '2025-12-21T06:02:39', '{"marker": "U", "source": "secondary"}'), ('SYS-0191', '2025-12-19T06:17:42', '{"marker": "W", "source": "secondary"}'), ('SYS-0182', '2025-12-19T06:17:33', '{"marker": "O", "source": "secondary"}'), ('SYS-0195', '2025-12-19T06:17:46', '{"marker": "D", "source": "secondary"}'), ('SYS-0227', '2025-12-19T06:23:27', '{"marker": "A", "source": "secondary"}'), ('SYS-0032', '2025-12-23T15:55:31', '{"marker": "I", "source": "secondary"}'), ('SYS-0177', '2025-12-19T06:17:31', '{"marker": "D", "source": "secondary"}'), ('SYS-0068', '2025-12-23T06:02:14', '{"marker": "G", "source": "secondary"}'), ('SYS-0069', '2025-12-23T06:02:15', '{"marker": "H", "source": "secondary"}'), ('SYS-0009', '2025-12-23T06:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0176', '2025-12-19T06:17:30', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-23T06:02:07', '{"marker": "B", "source": "primary"}'), ('SYS-0099', '2025-12-21T06:02:44', '{"marker": "E", "source": "secondary"}'), ('SYS-0014', '2025-12-23T15:55:13', '{"marker": "S", "source": "secondary"}'), ('SYS-0042', '2025-12-23T15:55:41', '{"marker": "S", "source": "secondary"}'), ('SYS-0006', '2025-12-24T08:02:05', '{"marker": "R", "source": "primary"}'), ('SYS-0187', '2025-12-19T06:17:38', '{"marker": "A", "source": "secondary"}'), ('SYS-0174', '2025-12-19T06:17:28', '{"marker": "U", "source": "secondary"}'), ('SYS-0056', '2025-12-23T15:55:55', '{"marker": "A", "source": "secondary"}'), ('SYS-0199', '2025-12-19T06:17:50', '{"marker": "E", "source": "secondary"}'), ('SYS-0120', '2025-12-19T06:03:04', '{"marker": "N", "source": "secondary"}'), ('SYS-0003', '2025-12-24T08:02:02', '{"marker": "Z", "source": "primary"}'), ('SYS-0116', '2025-12-19T06:02:59', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-24T18:10:18', '{"marker": "U", "source": "secondary"}'), ('SYS-0205', '2025-12-19T06:23:05', '{"marker": "Q", "source": "secondary"}'), ('SYS-0006', '2025-12-24T08:10:05', '{"marker": "T", "source": "primary"}'), ('SYS-0145', '2025-12-19T06:03:29', '{"marker": "Q", "source": "secondary"}'), ('SYS-0004', '2025-12-24T08:10:03', '{"marker": "E", "source": "primary"}'), ('SYS-0065', '2025-12-23T06:02:11', '{"marker": "Q", "source": "secondary"}'), ('SYS-0086', '2025-12-21T06:02:31', '{"marker": "L", "source": "secondary"}'), ('SYS-0188', '2025-12-19T06:17:39', '{"marker": "G", "source": "secondary"}'), ('SYS-0183', '2025-12-19T06:17:34', '{"marker": "F", "source": "secondary"}'), ('SYS-0062', '2025-12-23T06:02:08', '{"marker": "O", "source": "secondary"}'), ('SYS-0058', '2025-12-23T15:55:57', '{"marker": "V", "source": "secondary"}'), ('SYS-0169', '2025-12-19T06:17:23', '{"marker": "H", "source": "secondary"}'), ('SYS-0105', '2025-12-19T06:02:49', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-24T18:10:11', '{"marker": "A", "source": "secondary"}'), ('SYS-0005', '2025-12-23T06:02:04', '{"marker": "P", "source": "primary"}'), ('SYS-0131', '2025-12-19T06:03:15', '{"marker": "W", "source": "secondary"}'), ('SYS-0232', '2025-12-19T06:23:32', '{"marker": "K", "source": "secondary"}'), ('SYS-0028', '2025-12-23T15:55:27', '{"marker": "L", "source": "secondary"}'), ('SYS-0070', '2025-12-23T06:02:16', '{"marker": "I", "source": "secondary"}'), ('SYS-0029', '2025-12-23T15:55:28', '{"marker": "A", "source": "secondary"}'), ('SYS-0127', '2025-12-19T06:03:11', '{"marker": "A", "source": "secondary"}'), ('SYS-0010', '2025-12-24T18:10:09', '{"marker": "Q", "source": "secondary"}'), ('SYS-0025', '2025-12-23T15:55:24', '{"marker": "I", "source": "secondary"}'), ('SYS-0164', '2025-12-19T06:17:18', '{"marker": "S", "source": "secondary"}'), ('SYS-0114', '2025-12-19T06:02:58', '{"marker": "U", "source": "secondary"}'), ('SYS-0009', '2025-12-24T18:10:08', '{"marker": "S", "source": "secondary"}'), ('SYS-0184', '2025-12-19T06:17:35', '{"marker": "S", "source": "secondary"}'), ('SYS-0003', '2025-12-24T18:10:02', '{"marker": "V", "source": "secondary"}'), ('SYS-0053', '2025-12-23T15:55:35', '{"marker": "I", "source": "secondary"}'), ('SYS-0117', '2025-12-19T06:03:01', '{"marker": "D", "source": "secondary"}'), ('SYS-0074', '2025-12-23T06:02:20', '{"marker": "U", "source": "secondary"}'), ('SYS-0198', '2025-12-19T06:17:49', '{"marker": "V", "source": "secondary"}'), ('SYS-0036', '2025-12-23T15:55:35', '{"marker": "U", "source": "secondary"}'), ('SYS-0112', '2025-12-19T06:02:56', '{"marker": "K", "source": "secondary"}'), ('SYS-0023', '2025-12-23T15:55:22', '{"marker": "C", "source": "secondary"}'), ('SYS-0121', '2025-12-19T06:03:05', '{"marker": "T", "source": "secondary"}'), ('SYS-0181', '2025-12-19T06:17:32', '{"marker": "T", "source": "secondary"}'), ('SYS-0003', '2025-12-24T18:10:13', '{"marker": "H", "source": "secondary"}'), ('SYS-0130', '2025-12-19T06:03:14', '{"marker": "I", "source": "secondary"}'), ('SYS-0066', '2025-12-23T06:02:12', '{"marker": "L", "source": "secondary"}'), ('SYS-0203', '2025-12-19T06:17:54', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-24T18:10:01', '{"marker": "D", "source": "secondary"}'), ('SYS-0104', '2025-12-19T06:02:48', '{"marker": "S", "source": "secondary"}'), ('SYS-0017', '2025-12-23T15:55:16', '{"marker": "A", "source": "secondary"}'), ('SYS-0004', '2025-12-24T18:10:03', '{"marker": "E", "source": "secondary"}'), ('SYS-0238', '2025-12-19T06:23:38', '{"marker": "D", "source": "secondary"}'), ('SYS-0110', '2025-12-19T06:02:54', '{"marker": "I", "source": "secondary"}'), ('SYS-0071', '2025-12-23T06:02:17', '{"marker": "W", "source": "secondary"}'), ('SYS-0004', '2025-12-24T18:10:14', '{"marker": "I", "source": "secondary"}'), ('SYS-0031', '2025-12-23T15:55:30', '{"marker": "H", "source": "secondary"}'), ('SYS-0004', '2025-12-23T06:02:03', '{"marker": "M", "source": "primary"}'), ('SYS-0141', '2025-12-19T06:03:25', '{"marker": "T", "source": "secondary"}'), ('SYS-0207', '2025-12-19T06:23:07', '{"marker": "A", "source": "secondary"}'), ('SYS-0122', '2025-12-19T06:03:06', '{"marker": "O", "source": "secondary"}'), ('SYS-0006', '2025-12-24T18:10:05', '{"marker": "T", "source": "secondary"}'), ('SYS-0030', '2025-12-23T15:55:29', '{"marker": "G", "source": "secondary"}'), ('SYS-0192', '2025-12-19T06:17:43', '{"marker": "K", "source": "secondary"}'), ('SYS-0144', '2025-12-19T06:03:28', '{"marker": "S", "source": "secondary"}'), ('SYS-0098', '2025-12-21T06:02:43', '{"marker": "V", "source": "secondary"}'), ('SYS-0200', '2025-12-19T06:17:51', '{"marker": "N", "source": "secondary"}'), ('SYS-0007', '2025-12-24T08:02:06', '{"marker": "K", "source": "primary"}'), ('SYS-0220', '2025-12-19T06:23:20', '{"marker": "N", "source": "secondary"}'), ('SYS-0063', '2025-12-23T06:02:09', '{"marker": "F", "source": "secondary"}'), ('SYS-0109', '2025-12-19T06:02:53', '{"marker": "H", "source": "secondary"}'), ('SYS-0179', '2025-12-19T06:17:30', '{"marker": "E", "source": "secondary"}'), ('SYS-0163', '2025-12-19T06:17:17', '{"marker": "F", "source": "secondary"}'), ('SYS-0162', '2025-12-19T06:17:16', '{"marker": "O", "source": "secondary"}'), ('SYS-0222', '2025-12-19T06:23:22', '{"marker": "O", "source": "secondary"}'), ('SYS-0134', '2025-12-19T06:03:18', '{"marker": "U", "source": "secondary"}'), ('SYS-0004', '2025-12-24T08:02:03', '{"marker": "M", "source": "primary"}'), ('SYS-0139', '2025-12-19T06:03:23', '{"marker": "E", "source": "secondary"}'), ('SYS-0097', '2025-12-21T06:02:42', '{"marker": "D", "source": "secondary"}'), ('SYS-0146', '2025-12-19T06:03:30', '{"marker": "L", "source": "secondary"}'), ('SYS-0101', '2025-12-19T06:02:45', '{"marker": "T", "source": "secondary"}'), ('SYS-0201', '2025-12-19T06:17:52', '{"marker": "T", "source": "secondary"}'), ('SYS-0051', '2025-12-23T15:55:33', '{"marker": "C", "source": "secondary"}'), ('SYS-0172', '2025-12-19T06:17:26', '{"marker": "K", "source": "secondary"}'), ('SYS-0083', '2025-12-21T06:02:28', '{"marker": "F", "source": "secondary"}'), ('SYS-0009', '2025-12-24T08:10:08', '{"marker": "S", "source": "primary"}'), ('SYS-0124', '2025-12-19T06:03:08', '{"marker": "S", "source": "secondary"}'), ('SYS-0041', '2025-12-23T15:55:40', '{"marker": "F", "source": "secondary"}'), ('SYS-0049', '2025-12-23T15:55:31', '{"marker": "W", "source": "secondary"}'), ('SYS-0202', '2025-12-19T06:17:53', '{"marker": "O", "source": "secondary"}'), ('SYS-0178', '2025-12-19T06:17:29', '{"marker": "V", "source": "secondary"}'), ('SYS-0090', '2025-12-21T06:02:35', '{"marker": "I", "source": "secondary"}'), ('SYS-0006', '2025-12-23T06:02:05', '{"marker": "R", "source": "primary"}'), ('SYS-0073', '2025-12-23T06:02:19', '{"marker": "C", "source": "secondary"}'), ('SYS-0076', '2025-12-23T06:02:22', '{"marker": "A", "source": "secondary"}'), ('SYS-0217', '2025-12-19T06:23:17', '{"marker": "D", "source": "secondary"}'), ('SYS-0159', '2025-12-19T06:17:13', '{"marker": "E", "source": "secondary"}'), ('SYS-0161', '2025-12-19T06:17:15', '{"marker": "T", "source": "secondary"}'), ('SYS-0221', '2025-12-19T06:23:21', '{"marker": "T", "source": "secondary"}'), ('SYS-0236', '2025-12-19T06:23:36', '{"marker": "U", "source": "secondary"}'), ('SYS-0064', '2025-12-23T06:02:10', '{"marker": "S", "source": "secondary"}'), ('SYS-0126', '2025-12-19T06:03:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0024', '2025-12-23T15:55:23', '{"marker": "H", "source": "secondary"}'), ('SYS-0113', '2025-12-19T06:02:57', '{"marker": "C", "source": "secondary"}'), ('SYS-0038', '2025-12-23T15:55:37', '{"marker": "L", "source": "secondary"}'), ('SYS-0075', '2025-12-23T06:02:21', '{"marker": "D", "source": "secondary"}'), ('SYS-0088', '2025-12-21T06:02:33', '{"marker": "G", "source": "secondary"}'), ('SYS-0007', '2025-12-24T08:10:06', '{"marker": "O", "source": "primary"}'), ('SYS-0067', '2025-12-23T06:02:13', '{"marker": "A", "source": "secondary"}'), ('SYS-0156', '2025-12-19T06:17:10', '{"marker": "A", "source": "secondary"}'), ('SYS-0040', '2025-12-23T15:55:39', '{"marker": "O", "source": "secondary"}'), ('SYS-0219', '2025-12-19T06:23:19', '{"marker": "E", "source": "secondary"}'), ('SYS-0166', '2025-12-19T06:17:20', '{"marker": "L", "source": "secondary"}'), ('SYS-0007', '2025-12-24T18:10:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0153', '2025-12-19T06:17:07', '{"marker": "C", "source": "secondary"}'), ('SYS-0077', '2025-12-23T06:02:23', '{"marker": "D", "source": "secondary"}'), ('SYS-0084', '2025-12-21T06:02:29', '{"marker": "S", "source": "secondary"}'); INSERT INTO incoming_dispatches (system_id, dispatched_at, payload) VALUES ('SYS-0013', '2025-12-23T15:55:12', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-24T08:10:01', '{"marker": "D", "source": "primary"}'), ('SYS-0012', '2025-12-23T15:55:11', '{"marker": "O", "source": "secondary"}'), ('SYS-0108', '2025-12-19T06:02:52', '{"marker": "G", "source": "secondary"}'), ('SYS-0165', '2025-12-19T06:17:19', '{"marker": "Q", "source": "secondary"}'), ('SYS-0189', '2025-12-19T06:17:40', '{"marker": "H", "source": "secondary"}'), ('SYS-0050', '2025-12-23T15:55:32', '{"marker": "K", "source": "secondary"}'), ('SYS-0138', '2025-12-19T06:03:22', '{"marker": "V", "source": "secondary"}'), ('SYS-0087', '2025-12-21T06:02:32', '{"marker": "A", "source": "secondary"}'), ('SYS-0208', '2025-12-19T06:23:08', '{"marker": "G", "source": "secondary"}'), ('SYS-0057', '2025-12-23T15:55:56', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-24T18:10:12', '{"marker": "G", "source": "secondary"}'), ('SYS-0011', '2025-12-24T08:10:10', '{"marker": "L", "source": "primary"}'), ('SYS-0226', '2025-12-19T06:23:26', '{"marker": "L", "source": "secondary"}'), ('SYS-0206', '2025-12-19T06:23:06', '{"marker": "L", "source": "secondary"}'), ('SYS-0118', '2025-12-19T06:03:02', '{"marker": "V", "source": "secondary"}'), ('SYS-0185', '2025-12-19T06:17:36', '{"marker": "Q", "source": "secondary"}'), ('SYS-0218', '2025-12-19T06:23:18', '{"marker": "V", "source": "secondary"}'), ('SYS-0018', '2025-12-23T15:55:17', '{"marker": "G", "source": "secondary"}'), ('SYS-0005', '2025-12-24T18:10:04', '{"marker": "N", "source": "secondary"}'), ('SYS-0093', '2025-12-21T06:02:38', '{"marker": "C", "source": "secondary"}'), ('SYS-0011', '2025-12-24T18:10:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0011', '2025-12-23T15:55:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0143', '2025-12-19T06:03:27', '{"marker": "F", "source": "secondary"}'), ('SYS-0152', '2025-12-19T06:17:06', '{"marker": "K", "source": "secondary"}'), ('SYS-0015', '2025-12-23T15:55:14', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-23T06:02:00', '{"marker": "X", "source": "primary"}'), ('SYS-0209', '2025-12-19T06:23:09', '{"marker": "H", "source": "secondary"}'), ('SYS-0133', '2025-12-19T06:03:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0158', '2025-12-19T06:17:12', '{"marker": "V", "source": "secondary"}'), ('SYS-0195', '2025-12-19T06:17:46', '{"marker": "D", "source": "secondary"}'), ('SYS-0182', '2025-12-19T06:17:33', '{"marker": "O", "source": "secondary"}'), ('SYS-0033', '2025-12-23T15:55:32', '{"marker": "W", "source": "secondary"}'), ('SYS-0083', '2025-12-21T06:02:28', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-23T06:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0076', '2025-12-23T06:02:22', '{"marker": "A", "source": "secondary"}'), ('SYS-0009', '2025-12-24T08:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0194', '2025-12-19T06:17:45', '{"marker": "U", "source": "secondary"}'); ``` ``` $ sqlite3 SQLite version 3.50.4 2025-07-30 19:33:53 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day15-inserts.sql Parse error near line 19: unrecognized token: ":" 6:02:26', '{"marker": "T", "source": "secondary"}'::jsonb), ('SYS-0137', '2025 error here ---^ Parse error near line 275: unrecognized token: ":" 5:55:12', '{"marker": "F", "source": "secondary"}'::jsonb), ('SYS-0002', '2025 error here ---^ sqlite> .read day15-inserts-sqlite.sql sqlite> .schema CREATE TABLE system_dispatches ( id SERIAL PRIMARY KEY, system_id TEXT NOT NULL, dispatched_at TIMESTAMP NOT NULL, payload JSONB NOT NULL, marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, UNIQUE (system_id, dispatched_at, payload) ); CREATE TABLE incoming_dispatches ( system_id TEXT, dispatched_at TIMESTAMP, payload JSONB ); sqlite> .mode table sqlite> SELECT * FROM system_dispatches LIMIT 10; +----+-----------+---------------------+----------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+----------------------------------------+---------------+ | | SYS-0081 | 2025-12-21T06:02:26 | {"marker": "T", "source": "secondary"} | T | | | SYS-0137 | 2025-12-19T06:03:21 | {"marker": "D", "source": "secondary"} | D | | | SYS-0237 | 2025-12-19T06:23:37 | {"marker": "D", "source": "secondary"} | D | | | SYS-0006 | 2025-12-24T18:10:16 | {"marker": "K", "source": "secondary"} | K | | | SYS-0170 | 2025-12-19T06:17:24 | {"marker": "I", "source": "secondary"} | I | | | SYS-0224 | 2025-12-19T06:23:24 | {"marker": "S", "source": "secondary"} | S | | | SYS-0007 | 2025-12-24T18:10:06 | {"marker": "O", "source": "secondary"} | O | | | SYS-0035 | 2025-12-23T15:55:34 | {"marker": "C", "source": "secondary"} | C | | | SYS-0010 | 2025-12-23T15:55:09 | {"marker": "L", "source": "secondary"} | L | | | SYS-0037 | 2025-12-23T15:55:36 | {"marker": "D", "source": "secondary"} | D | +----+-----------+---------------------+----------------------------------------+---------------+ sqlite> SELECT * FROM incoming_dispatches LIMIT 10; +-----------+---------------------+----------------------------------------+ | system_id | dispatched_at | payload | +-----------+---------------------+----------------------------------------+ | SYS-0013 | 2025-12-23T15:55:12 | {"marker": "F", "source": "secondary"} | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | | SYS-0012 | 2025-12-23T15:55:11 | {"marker": "O", "source": "secondary"} | | SYS-0108 | 2025-12-19T06:02:52 | {"marker": "G", "source": "secondary"} | | SYS-0165 | 2025-12-19T06:17:19 | {"marker": "Q", "source": "secondary"} | | SYS-0189 | 2025-12-19T06:17:40 | {"marker": "H", "source": "secondary"} | | SYS-0050 | 2025-12-23T15:55:32 | {"marker": "K", "source": "secondary"} | | SYS-0138 | 2025-12-19T06:03:22 | {"marker": "V", "source": "secondary"} | | SYS-0087 | 2025-12-21T06:02:32 | {"marker": "A", "source": "secondary"} | | SYS-0208 | 2025-12-19T06:23:08 | {"marker": "G", "source": "secondary"} | +-----------+---------------------+----------------------------------------+ sqlite> ``` Good to go! ## Problem > Reconstruct the final confirmation phrase to text Santa based on the elves’ hazy recollection of how they solved this problem before. > > Your final result should include the marker_letter for each system, using only the most recent dispatch from a primary source. Once the correct dispatch has been identified for every system, combine the results and order them by dispatched_at in ascending order to reveal the confirmation phrase. > > The sleigh won’t launch without it. So, we need to first make sure we have the right `dispatches`. We have two tables. 1. `system_dispatches` 2. `incoming_dispatches` The problem statement also states: > The system kept throwing errors until we figured out how to handle duplicates. Whatever you do the records already in system_dispatches must take precedence.” So, we need to take the `system_dispatches` as the ground truth. However we need to include something or the other from the `incoming_dispatches` as there are new entries being created. So, we will first try to insert into the system dispatches everything in the `incoming_dispatches` ```sql INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; ``` ``` sqlite> INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; Runtime error: UNIQUE constraint failed: system_dispatches.system_id, system_dispatches.dispatched_at, system_dispatches.payload (19) sqlite> ``` Ops! ```sql SELECT COUNT(*) as system_dispatches_count FROM system_dispatches ; SELECT COUNT(*) as incoming_dispatches_count FROM incoming_dispatches; SELECT COUNT(*) as common_dispatches_count FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches INTERSECT SELECT * FROM incoming_dispatches); ``` ``` sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 254 | +----------+ sqlite> SELECT COUNT(*) FROM incoming_dispatches; +----------+ | COUNT(*) | +----------+ | 38 | +----------+ sqlite> SELECT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches UNION SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> SELECT DISTINCT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches UNION SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> SELECT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches INTERSECT SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 10 | +----------+ sqlite> ``` We need to be careful about what we insert into the `system_dispatches`. I think the duplicates causes this conflict in constraints, we can ignore on the constraint failures and don't insert that row. ```sql INSERT OR IGNORE INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; ``` Ok, now if we see the count, we get `282` which is making sense. - The original `system_dispatches` had a count of `254` before merge. - The original `incoming_dispatches` had a count of `38`. - After merging `incoming_dispatches` into `system_dispatches` table, we have a `282` records in `system_dispatches` but `254+38=292`, what about the `10`? Those were already there in the `system_dispatches`. As we saw in the earlier query, there were `10` rows in common from the `incoming_dispatches` table before merge. ``` sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 254 | +----------+ sqlite> SELECT COUNT(*) FROM incoming_dispatches; +----------+ | COUNT(*) | +----------+ | 38 | +----------+ sqlite> INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; Runtime error: UNIQUE constraint failed: system_dispatches.system_id, system_dispatches.dispatched_at, system_dispatches.payload (19) sqlite> INSERT OR IGNORE INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> ``` Now, to the actual problem. What we need to do now? - Using only the most recent dispatch from a primary source. - Once the correct dispatch has been identified for every system, combine the results and order them by dispatched_at in ascending order - Reveal the confirmation phrase We have the single source now! ```sql SELECT * FROM system_dispatches LIMIT 10; ``` ``` SELECT * FROM system_dispatches LIMIT 10; +----+-----------+---------------------+----------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+----------------------------------------+---------------+ | | SYS-0081 | 2025-12-21T06:02:26 | {"marker": "T", "source": "secondary"} | T | | | SYS-0137 | 2025-12-19T06:03:21 | {"marker": "D", "source": "secondary"} | D | | | SYS-0237 | 2025-12-19T06:23:37 | {"marker": "D", "source": "secondary"} | D | | | SYS-0006 | 2025-12-24T18:10:16 | {"marker": "K", "source": "secondary"} | K | | | SYS-0170 | 2025-12-19T06:17:24 | {"marker": "I", "source": "secondary"} | I | | | SYS-0224 | 2025-12-19T06:23:24 | {"marker": "S", "source": "secondary"} | S | | | SYS-0007 | 2025-12-24T18:10:06 | {"marker": "O", "source": "secondary"} | O | | | SYS-0035 | 2025-12-23T15:55:34 | {"marker": "C", "source": "secondary"} | C | | | SYS-0010 | 2025-12-23T15:55:09 | {"marker": "L", "source": "secondary"} | L | | | SYS-0037 | 2025-12-23T15:55:36 | {"marker": "D", "source": "secondary"} | D | +----+-----------+---------------------+----------------------------------------+---------------+ sqlite> ``` We have 4 columns. - system_id - dispatched_at - payload (json string) - marker (its a generated table from the payload["marker"] string) So, we need to group by system, that's where the `system_id` is for. Then we need to order by `dispatched_at` with the latest ones. Finally we also need to filter the records that has `payload["source"]` as `primary`. But wait, if we group by `system_id` then how can we get the latest dispatched ordered correctly? We can't use it in the `ORDER BY` as the group would have already been created right? Oh! We might need window functions, but just a moment, can we do it without them? Let's try. ```sql SELECT system_id, MAX(dispatched_at), payload, marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` Oh boy, that was simple! We just grouped by `system_id`, filtered the source as `primary` and selected the row with `MAX(dispatched_at)` which will give us the latest dispatch record for a system. Boom! The confirmation phrase is `ADVENTOFSQL` We can just select the `marker_letter` ```sql SELECT marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` ``` sqlite> SELECT marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; +---------------+ | marker_letter | +---------------+ | X | | Y | | Z | | M | | P | | R | | K | | B | | U | | C | | J | +---------------+ sqlite> ``` Whops! What happened, the marker letter changed? Yep, because we have ordered by `dispatched_at DESC` but never told how to return the rows, by using `MAX(dispatched_at)` we were selecting the latest date record. Without it, we are selecting the first record in the group that could be the first or the oldest dispatched_at time. So, we need `MAX(dispatched_at)` included in the selected result set. ```sql SELECT marker_letter, MAX(dispatched_at) FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` ``` SELECT marker_letter, MAX(dispatched_at) FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; +---------------+---------------------+ | marker_letter | MAX(dispatched_at) | +---------------+---------------------+ | A | 2025-12-24T08:10:00 | | D | 2025-12-24T08:10:01 | | V | 2025-12-24T08:10:02 | | E | 2025-12-24T08:10:03 | | N | 2025-12-24T08:10:04 | | T | 2025-12-24T08:10:05 | | O | 2025-12-24T08:10:06 | | F | 2025-12-24T08:10:07 | | S | 2025-12-24T08:10:08 | | Q | 2025-12-24T08:10:09 | | L | 2025-12-24T08:10:10 | +---------------+---------------------+ sqlite> ``` There, we go! It scared me for a moment. We order by, only keep the `source` as `primary` and then select the latest `dispatched_at` for a given system_id. If you don't want to do this way! I found a few more hacks ### With Subquery We can SELECT the `MAX(dispatched_at)` in a subquery for that system_id and filter based on the `primary` source as usual. ```sql SELECT * FROM system_dispatches WHERE json_extract(system_dispatches.payload, '$.source') = 'primary' AND system_dispatches.dispatched_at = ( SELECT MAX(latest_dispatches.dispatched_at) FROM system_dispatches latest_dispatches WHERE latest_dispatches.system_id = system_dispatches.system_id AND json_extract(latest_dispatches.payload, '$.source') = 'primary' ) ORDER BY system_dispatches.dispatched_at ASC; ``` ``` sqlite> SELECT * FROM system_dispatches WHERE json_extract(system_dispatches.payload, '$.source') = 'primary' AND system_dispatches.dispatched_at = ( SELECT MAX(latest_dispatches.dispatched_at) FROM system_dispatches latest_dispatches WHERE latest_dispatches.system_id = system_dispatches.system_id AND json_extract(latest_dispatches.payload, '$.source') = 'primary' ) ORDER BY system_dispatches.dispatched_at ASC; +----+-----------+---------------------+--------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+--------------------------------------+---------------+ | | SYS-0001 | 2025-12-24T08:10:00 | {"marker": "A", "source": "primary"} | A | | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | D | | | SYS-0003 | 2025-12-24T08:10:02 | {"marker": "V", "source": "primary"} | V | | | SYS-0004 | 2025-12-24T08:10:03 | {"marker": "E", "source": "primary"} | E | | | SYS-0005 | 2025-12-24T08:10:04 | {"marker": "N", "source": "primary"} | N | | | SYS-0006 | 2025-12-24T08:10:05 | {"marker": "T", "source": "primary"} | T | | | SYS-0007 | 2025-12-24T08:10:06 | {"marker": "O", "source": "primary"} | O | | | SYS-0008 | 2025-12-24T08:10:07 | {"marker": "F", "source": "primary"} | F | | | SYS-0009 | 2025-12-24T08:10:08 | {"marker": "S", "source": "primary"} | S | | | SYS-0010 | 2025-12-24T08:10:09 | {"marker": "Q", "source": "primary"} | Q | | | SYS-0011 | 2025-12-24T08:10:10 | {"marker": "L", "source": "primary"} | L | +----+-----------+---------------------+--------------------------------------+---------------+ sqlite> ``` ### Window Funciton We can even use the window function to partition by the `system_id` and there itself order by the `dispatched_at` latest time. And then select it as a CTE table. We can use a `ROW_NUMBER` function to assign a rank to each row per system_id and ordered by the latest `dispatched_at` ```sql WITH latest_dispatches AS ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY system_id ORDER BY dispatched_at DESC ) as rank FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' ) SELECT * FROM latest_dispatches WHERE rank = 1 ORDER BY dispatched_at ASC; ``` The rest is the same as the above query. We order by dispatched latest but then order by ascending (oldest first) in the final result set. ``` sqlite> WITH latest_dispatches AS ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY system_id ORDER BY dispatched_at DESC ) as rank FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' ) SELECT * FROM latest_dispatches WHERE rank = 1 ORDER BY dispatched_at ASC; +----+-----------+---------------------+--------------------------------------+---------------+------+ | id | system_id | dispatched_at | payload | marker_letter | rank | +----+-----------+---------------------+--------------------------------------+---------------+------+ | | SYS-0001 | 2025-12-24T08:10:00 | {"marker": "A", "source": "primary"} | A | 1 | | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | D | 1 | | | SYS-0003 | 2025-12-24T08:10:02 | {"marker": "V", "source": "primary"} | V | 1 | | | SYS-0004 | 2025-12-24T08:10:03 | {"marker": "E", "source": "primary"} | E | 1 | | | SYS-0005 | 2025-12-24T08:10:04 | {"marker": "N", "source": "primary"} | N | 1 | | | SYS-0006 | 2025-12-24T08:10:05 | {"marker": "T", "source": "primary"} | T | 1 | | | SYS-0007 | 2025-12-24T08:10:06 | {"marker": "O", "source": "primary"} | O | 1 | | | SYS-0008 | 2025-12-24T08:10:07 | {"marker": "F", "source": "primary"} | F | 1 | | | SYS-0009 | 2025-12-24T08:10:08 | {"marker": "S", "source": "primary"} | S | 1 | | | SYS-0010 | 2025-12-24T08:10:09 | {"marker": "Q", "source": "primary"} | Q | 1 | | | SYS-0011 | 2025-12-24T08:10:10 | {"marker": "L", "source": "primary"} | L | 1 | +----+-----------+---------------------+--------------------------------------+---------------+------+ sqlite> ``` Phew! That was a good one to end the advent of SQL! I enjoyed it! I learnt a ton - CTEs - JOINs (some wired stuff can be done) - Window Functions (LAG, LEAD, ROW_NUMBER) - FTS (in SQLite) - Json parsing - date shenanigans - CTEs don't support insert and delete in SQLite (it ruined my day 10 solution) - Recursive CTEs - String manipulation (thanks to xml parsing) - I need to write a post about explaining what I learned. Thanks [Aaron francis](https://aaronfrancis.com/) from [databaseschool](https://databaseschool.com/) for this challenge and explaining in depth on each day(i didn't watch all) and [Kelsey Petrich](https://x.com/krpetrich) for the lore of each problem, those were really lovely to read! Happy Coding :) Merry Christmas Happy New year Whatever you celebrate!

Advent of SQL 2025 Day 14: Ski Resort Paths

Sun, 28 Dec 2025 00:00:00 UTC

## Advent of SQL, Day 14 - Ski Resort Paths Ok, almost to the penultimate day in the series. It is day 14 of Advent of SQL. Let's grab the SQL for the day. ```sql DROP TABLE IF EXISTS mountain_network; CREATE TABLE mountain_network ( id INTEGER PRIMARY KEY, from_node TEXT, to_node TEXT, node_type TEXT, -- 'Lift' or 'Trail' difficulty TEXT -- Only applicable for trails: 'green', 'blue', 'black', 'double_black' ); INSERT INTO mountain_network (id, from_node, to_node, node_type, difficulty) VALUES (1, 'Outlaw Express', 'Stairway Lift', 'Lift', NULL), (2, 'Outlaw Express', 'Top Gun Bowl', 'Trail', 'black'), (3, 'Top Gun Bowl', 'Top Gun', 'Trail', 'black'), (4, 'Top Gun', 'Montoya', 'Trail', 'blue'), (5, 'Montoya', 'Center Aisle', 'Trail', 'green'), (6, 'Center Aisle', 'Lower Stampede', 'Trail', 'green'), (7, 'Stairway Lift', 'Red''s Lift', 'Lift', NULL), (8, 'Stairway Lift', 'Broadway', 'Trail', 'green'), (9, 'Red''s Lift', 'Bearclaw', 'Trail', 'blue'), (10, 'Bearclaw', 'Last Chance', 'Trail', 'blue'), (11, 'Last Chance', 'Diamondback', 'Trail', 'blue'), (12, 'Diamondback', 'Broadway', 'Trail', 'green'), (13, 'Red''s Lift', 'Bishop''s Bowl', 'Trail', 'black'), (14, 'Red''s Lift', 'Amy''s Ridge', 'Trail', 'blue'), (15, 'Amy''s Ridge', 'Grizzly Bowl', 'Trail', 'black'), (16, 'Flathead Lift', 'Amy''s Ridge', 'Trail', 'blue'), (17, 'Jake''s Lift', 'Wildwood Lift', 'Lift', NULL), (18, 'Wildwood Lift', 'Sidewinder', 'Trail', 'green'), (19, 'Wildwood Lift', 'Brightside', 'Trail', 'blue'), (20, 'Brightside', 'Moonrise', 'Trail', 'green'), (21, 'Moonrise', 'Draw', 'Trail', 'green'), (22, 'Moonrise', 'Lone Pine', 'Trail', 'blue'), (23, 'Draw', 'Maverick', 'Trail', 'blue'), (24, 'Draw', 'Broadway', 'Trail', 'green'), (25, 'Broadway', 'Outlaw Trail', 'Trail', 'green'), (26, 'Outlaw Trail', 'Center Aisle', 'Trail', 'green'), (27, 'Center Aisle', 'Bandit', 'Trail', 'green'), (28, 'Jake''s Lift', 'Maverick', 'Trail', 'blue'); ``` That's it, we just have 1 table called `mountain_network` and it has `28` records. Let's look at the problem statement to check what we need to do to make sense of those 28 rows. ## Problem > Find all the possible routes from Jake's Lift to Maverick. None of the possible routes will take more than 12 connections. Oh! This is a graph like or network like problem. Ouch! Relational databases looks like they would be a good fit for these stuff, but if the data is just in the single table, there's not really much help. Let's see how we can think about it. ### JOINs and UNIONS We need to find all the routes from a start node(record) i.e. `Jake's Lift` and find all ways that lead to the node(record) i.e. `Maverick`. The table gives a list of edges i.e. from which node to which node there is a way or a trail or lift. So, we can do a simple select to check if the `from_node` has `Jake's Lift` and the `to_node` has `Maverick` right? But that would make it too long. Like we would have to look for 12 consecutive edges and branch off at each direction. For the first level it would look simple like this: ```sql SELECT mountain_network.from_node || ' -> ' || mountain_network.to_node AS path, 1 AS connections FROM mountain_network WHERE mountain_network.from_node = 'Jake''s Lift' AND mountain_network.to_node = 'Maverick'; ``` ``` sqlite> SELECT mountain_network.from_node || ' -> ' || mountain_network.to_node AS path, 1 AS connections FROM mountain_network WHERE mountain_network.from_node = 'Jake''s Lift' AND mountain_network.to_node = 'Maverick' ...> ; +-------------------------+-------------+ | path | connections | +-------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +-------------------------+-------------+ sqlite> ``` Luckily, we have one route directly from `Jake's Lift` to `Maverick`. But that might not be for all levels. Like we need to check then, from each of the start nodes that begin at `Jake's Lift`, then for all the nodes that begin at that then it branches of further till we have a dead end only at `Maverick`. Phew that's going to be a long one. ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node, 2 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift' AND T2.to_node = 'Maverick'; ``` That is the first branch off from `Jake's Lift`. If we remove the `T2.to_node = 'Maverick'` you would see all nodes from the `Jake's Lift`, not necessarily ending at `Maverick` ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node as path, 2 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift'; ``` ``` sqlite> SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node as path, 2 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift'; +--------------------------------------------+-------------+ | path | connections | +--------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside | 2 | | Jake's Lift -> Wildwood Lift -> Sidewinder | 2 | +--------------------------------------------+-------------+ sqlite> ``` Similarly we can do it for 3 and 4, and till 12 connections. ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node as path, 3 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift'; ``` ``` sqlite> SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node as path, 3 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift' ...> ; +--------------------------------------------------------+-------------+ | path | connections | +--------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise | 3 | +--------------------------------------------------------+-------------+ sqlite> ``` Yikes. That gives a long long query. I cannot write. ```sql SELECT T1.from_node || ' -> ' || T1.to_node AS path, 1 AS connections FROM mountain_network T1 WHERE T1.from_node = 'Jake''s Lift' AND T1.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node, 2 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift' AND T2.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node, 3 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift' AND T3.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node, 4 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node WHERE T1.from_node = 'Jake''s Lift' AND T4.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node, 5 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node WHERE T1.from_node = 'Jake''s Lift' AND T5.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node, 6 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node WHERE T1.from_node = 'Jake''s Lift' AND T6.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node, 7 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node WHERE T1.from_node = 'Jake''s Lift' AND T7.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node, 8 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node WHERE T1.from_node = 'Jake''s Lift' AND T8.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node, 9 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node WHERE T1.from_node = 'Jake''s Lift' AND T9.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node, 10 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node WHERE T1.from_node = 'Jake''s Lift' AND T10.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node || ' -> ' || T11.to_node, 11 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node JOIN mountain_network T11 ON T10.to_node = T11.from_node WHERE T1.from_node = 'Jake''s Lift' AND T11.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node || ' -> ' || T11.to_node || ' -> ' || T12.to_node, 12 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node JOIN mountain_network T11 ON T10.to_node = T11.from_node JOIN mountain_network T12 ON T11.to_node = T12.from_node WHERE T1.from_node = 'Jake''s Lift' AND T12.to_node = 'Maverick'; ``` ``` +--------------------------------------------------------------+-------------+ | path | connections | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | aw -> Maverick | | +--------------------------------------------------------------+-------------+ sqlite> ``` Ok, we get the result, but that looks like a terrifying query. Can we do better? I looked at the tutorial and we have something like `RECURSIVE CTE` Wow! This is quite challenging to explain. ### Recursive CTE We can create a recursive CTE that draws the path until we have not found the stop node that is, `Maverick` and the number of connections is not more than `12`. So here's how it goes. ```sql WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT full_path, connections FROM ski_paths WHERE to_node = 'Maverick' ORDER BY connections ASC; ``` We first define the base case i.e. what to select first. We get the first row where we start, the `Jake's Lift` and from there on we create a recursive select statement by referencing the CTE within it. The recursive bit is this one ```sql SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ``` We are referencing the CTE (table) within it as the recursive condition to select from the nodes where the end node or the `to_node` is not Maverick and connections haven't gone beyond `12`, we try to find the path. This would give us all the nodes. Let's see first, by checking the CTE as is. ```sql WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT * from ski_paths; ``` ``` sqlite> WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT * from ski_paths; +---------------+----------------+--------------------------------------------------------------+-------------+ | from_node | to_node | full_path | connections | +---------------+----------------+--------------------------------------------------------------+-------------+ | Jake's Lift | Wildwood Lift | Jake's Lift -> Wildwood Lift | 1 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Jake's Lift | Maverick | Jake's Lift -> Maverick | 1 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Wildwood Lift | Brightside | Jake's Lift -> Wildwood Lift -> Brightside | 2 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Wildwood Lift | Sidewinder | Jake's Lift -> Wildwood Lift -> Sidewinder | 2 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Brightside | Moonrise | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise | 3 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Moonrise | Draw | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 4 | | | | aw | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Moonrise | Lone Pine | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Lo | 4 | | | | ne Pine | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Draw | Broadway | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | | | aw -> Broadway | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Draw | Maverick | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | | | aw -> Maverick | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Broadway | Outlaw Trail | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 6 | | | | aw -> Broadway -> Outlaw Trail | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Outlaw Trail | Center Aisle | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 7 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Center Aisle | Bandit | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 8 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle -> Bandit | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Center Aisle | Lower Stampede | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 8 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle -> Lower Stam | | | | | pede | | +---------------+----------------+--------------------------------------------------------------+-------------+ sqlite> ``` You can see we have got quite a lot of paths, this are all the paths that start from `Jake's Lift` and have less than `12` connections. So, now we can simply filter with the `WHERE` clause in the case where the `to_node = 'Maverick'` and we would get the result. ``` sqlite> WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT full_path, connections FROM ski_paths WHERE to_node = 'Maverick' ORDER BY connections ASC; +--------------------------------------------------------------+-------------+ | full_path | connections | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | aw -> Maverick | | +--------------------------------------------------------------+-------------+ sqlite> ``` Ok, that gave the result correctly. And this is it! Sweet problem to learn about Recursive CTE! Off to the final day of advent of SQL 2025!

Advent of SQL 2025 Day 13: XML Travel Manifests

Sat, 27 Dec 2025 00:00:00 UTC

## Advent of SQL - Day 13, XML Travel Manifests Its day 13 of Advent of SQL, we have some xml to parse, which I don't think SQL can handle, but string manipulation to the rescue. Let's get the SQL for the day: ```sql DROP TABLE IF EXISTS travel_manifests; CREATE TABLE travel_manifests ( manifest_id INT PRIMARY KEY, vehicle_id TEXT, departure_time TIMESTAMP, manifest_xml XML ); INSERT INTO travel_manifests (manifest_id, vehicle_id, departure_time, manifest_xml) VALUES (1, 'SLEIGH-01', '2025-12-22 06:00:00', 'Nia GrantovernightSofia KimovernightignoredJonah Wolfestandardignored'), (2, 'SLEIGH-07', '2025-12-23 13:00:00', 'Ravi PatelpriorityKeiko ItostandardAnya PavlovstandardNia GrantpriorityignoredCarter LewispriorityignoredLayla Brooksstandardignored'), (3, 'FLIGHT-NP-9', '2025-12-22 18:00:00', 'Keiko ItoovernightJonah WolfepriorityDiego RamosovernightignoredPriya DaspriorityLayla Brooksstandard'), (4, 'TRAIN-ICE-3', '2025-12-22 18:00:00', 'lowIsla TorresovernightRavi PatelovernightHiro TanakastandardPriya Daspriority'), (5, 'FLIGHT-NP-9', '2025-12-22 17:00:00', 'Nia GrantpriorityMateo Cruzstandardignored'), (6, 'CARGO-12', '2025-12-22 15:00:00', 'Carter LewisovernightSofia KimpriorityHiro TanakapriorityLucas FordpriorityignoredNia Grantovernight'), (7, 'SLEIGH-01', '2025-12-22 11:00:00', 'unknownPriya DasstandardDiego RamosstandardLucas FordovernightCarter LewisovernightignoredHiro TanakapriorityignoredZara Sheikhovernight'), (8, 'CARGO-12', '2025-12-23 13:00:00', 'Jonah WolfestandardLayla BrooksovernightLeo BeckerovernightignoredRavi PatelstandardSofia KimpriorityignoredElena Moralesovernight'), (9, 'SLEIGH-01', '2025-12-23 10:00:00', 'Sofia KimovernightBianca PereirastandardZara SheikhovernightElena MoralesovernightignoredHiro TanakapriorityKeiko Itopriorityignored'), (10, 'SLEIGH-01', '2025-12-22 21:00:00', 'lowNia GrantstandardAva JohnsonstandardPriya DaspriorityMateo CruzstandardBianca PereiraovernightLeo Beckerovernight'), (11, 'SLEIGH-07', '2025-12-23 10:00:00', 'Jonah WolfepriorityBianca PereirapriorityAnya Pavlovstandard'), (12, 'SLEIGH-01', '2025-12-22 08:00:00', 'okRavi PatelpriorityignoredBianca PereirapriorityKeiko Itostandard'), (13, 'FLIGHT-NP-9', '2025-12-22 11:00:00', 'okNia GrantpriorityElena Moralesstandard'), (14, 'SLEIGH-01', '2025-12-22 14:00:00', 'highLayla BrooksstandardignoredAnya PavlovovernightKeiko ItostandardZara SheikhstandardignoredSofia Kimovernight'), (15, 'FLIGHT-NP-9', '2025-12-22 14:00:00', 'okPriya DaspriorityignoredBianca PereirastandardNia Grantstandard'); ``` Just one table, with some wild XML. Let's see what do we want to do in the problem statement. ## Problem > Using the `travel_manifests` table, extract the passenger information from the XML data and produce a report that shows all of the departure times for "CARGO" vehicles that have more than 20 passengers booked. Include in the results: > > - The vehicle_id > - The departure_time > - The total number of passengers on that departure > - Order the results by departure_time. Ok, so we need the number of passengers in the records which are of type `CARGO` and have more than 20 passengers booked. Interesting! Let's look at one record. ``` sqlite> .schema CREATE TABLE travel_manifests ( manifest_id INT PRIMARY KEY, vehicle_id TEXT, departure_time TIMESTAMP, manifest_xml XML ); sqlite> SELECT * FROM travel_manifests WHERE id = 1; Parse error: no such column: id SELECT * FROM travel_manifests WHERE id = 1; error here ---^ sqlite> SELECT * FROM travel_manifests LIMIT 1; +-------------+------------+---------------------+--------------------------------------------------------------+ | manifest_id | vehicle_id | departure_time | manifest_xml | +-------------+------------+---------------------+--------------------------------------------------------------+ | 1 | SLEIGH-01 | 2025-12-22 06:00:00 | Nia GrantovernightSofia KimovernightignoredJonah Wolfestandardignored | +-------------+------------+---------------------+--------------------------------------------------------------+ sqlite> ``` So, we have the following columns - `vehicle_id` which is I think related to filtering `CARGO` related vehicles only - `departure_time` which we just return as is - `manifest_xml`, oh! This is xml and it has passenger details. If we look carefully, we can see the xml looks like this: ```xml Nia Grant overnight Sofia Kim overnight ignored Jonah Wolfe standard ignored ``` We have `manifest` which has a property of `passengers` which is a list of `passenger` tags, inside of which, each element of `passenger` has its details like `name`, `ticket_class`, etc. We only want the count of `passengers`, how can we get that? The dirtiest way to do is to count the occurances of `` or `` in the xml string. We can do that with counting the full length of the xml string, and then dividing by the number of times the string can be replaced(which is the dirty part, there could be hidden `` string somewhere that might break this logic, but if it is a valid xml, it works). We count the number of characters left after we replace the string `` with empty string `''` so that we can get the difference of the total number of character and the number of characters occupied by the string ``. This difference if we divide by the length of `` will give us the count of the number of times the `` string is present in the xml string. Let's take an example from the above. The length of the xml string is 374. ```sql SELECT LENGTH(manifest_xml) FROM travel_manifests LIMIT 1; ``` ``` sqlite> SELECT LENGTH(manifest_xml) FROM travel_manifests LIMIT 1; +----------------------+ | LENGTH(manifest_xml) | +----------------------+ | 374 | +----------------------+ sqlite> ``` Let's replace the occurances of ` with empty string in the `manifest_xml` string, like so: ```sql SELECT REPLACE(manifest_xml, '', '') FROM travel_manifests LIMIT 1; ``` Now, we can see the string `` is gone from the returned result set. We can try getting its length now. ```sql SELECT LENGTH( REPLACE(manifest_xml, '', '') ) FROM travel_manifests LIMIT 1; ``` Now, it says `341` why? Because we removed (replaced with empty string) the occurences of ``. Let's get the length of `''` string, which should be `11` right? Spell it `p-a-s-s-e-n-g-e-r` as `pass` + `enger` (4+5=9) and 2 for `<>` so 11. Sometimes I don't know how to do math, I use SQL. ```sql SELECT LENGTH(''); ``` There it is `11`. Now, if you compute the difference of the actual length of XML with the removed parts of the `` what do we get? ```sql SELECT LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')) FROM travel_manifests LIMIT 1; ``` WE got `33`, why would you ask because `11` times 3 is `33`. We found three instances of ``, so we just need to divide by the length of `` or hard code it as `11` doesn't matter. We would get the number of occurances of `` which will give the number of passenger in the xml string. ```sql SELECT ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')) ) / LENGTH('') FROM travel_manifests LIMIT 1; ``` Phew, its `3`! That was a lot for a simple stuff. But hey its fun! ``` sqlite> SELECT (manifest_xml) FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | manifest_xml | +--------------------------------------------------------------+ | Nia GrantovernightSofia KimovernightignoredJonah Wolfestandardignored | +--------------------------------------------------------------+ sqlite> SELECT REPLACE(manifest_xml, '', '') FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | REPLACE(manifest_xml, '', '') | +--------------------------------------------------------------+ | Nia Grantov | | ernightSofia KimovernightignoredJonah Wolfe | | standardignored | +--------------------------------------------------------------+ sqlite> sqlite> SELECT LENGTH(REPLACE(manifest_xml, '', '')) FROM travel_manifests LIMIT 1; +--------------------------------------------------+ | LENGTH(REPLACE(manifest_xml, '', '')) | +--------------------------------------------------+ | 341 | +--------------------------------------------------+ sqlite> SELECT LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')) FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, ' SELECT LENGTH('') FROM travel_manifests LIMIT 1; +-----------------------+ | LENGTH('') | +-----------------------+ | 11 | +-----------------------+ sqlite> SELECT (LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')))/LENGTH('') FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | (LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, ' ``` Now, let construct the query to get the number of passengers. ```sql SELECT vehicle_id, departure_time, ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')) ) / LENGTH('') AS passengers_in_manifest FROM travel_manifests WHERE vehicle_id LIKE 'CARGO-%'; ``` We need to wrap it in a CTE to grab and group by the vehicle_id I believe, as there are similar entries. Also we need to group the records with the same departure time, so that we can combine the number of passengers for that vehicle. ```sql WITH passenger_counts AS ( SELECT vehicle_id, departure_time, ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '', '')) ) / LENGTH('') AS passengers_in_manifest FROM travel_manifests WHERE vehicle_id LIKE 'CARGO-%' ) SELECT vehicle_id, departure_time, SUM(passengers_in_manifest) AS total_passengers FROM passenger_counts GROUP BY vehicle_id, departure_time HAVING SUM(passengers_in_manifest) > 20 ORDER BY departure_time; ``` We count the number of passengers in the CTE and use it as a filter in the outer query as `SUM(passengers_in_manifest) > 20` which will give the right condition for us to get the result. We have to use `HAVING` as we need to do that after grouping the same `vehicle_id` and records across same `departure_time`. We also use the `vehicle_id LIKE 'CARGO-%'` in the CTE to filter it right at the inner query to avoid looping in all the queries for computing the number of passengers. That solves this problem. That's it from day 13 of Advent of SQL. There are other ways, but its the same parsing, We can use JOINs and stuff, but hey that was not the point of this. Anyways! See you tomorrow for day 14!

Advent of SQL 2025 Day 12: Archive Flight Records

Sat, 27 Dec 2025 00:00:00 UTC

## Advent of SQL - Day 12, Archive Flight Records We are on Day 12! Phew its almost done! Just 3 days more! Let's get the SQL! ```sql DROP TABLE IF EXISTS archive_records; CREATE TABLE archive_records ( id INT PRIMARY KEY, title TEXT, description TEXT ); INSERT INTO archive_records (id, title, description) VALUES (1, 'Flight Stabilization Prototype Analysis', 'This report details the latest advancements in stabilizing aerial maneuvers for enchanted sleighs. Initial tests yielded promising results, showcasing a marked decrease in turbulence during airborne navigation.'), (2, 'Lift Calibration Incident Log', 'During a routine lift calibration test, the sleigh experienced an unexpected upward surge, causing it to hover dangerously close to the workshop ceiling. Subsequent analysis revealed a miscalculation in the weight distribution formula, prompting a thorough review of all aerodynamic coefficients.'), (3, 'Aerial Aspirations: The Great Flop', 'Despite the initial excitement surrounding the design of our feather-laden airborne contraption, the prototype proved less than buoyant. The unexpected descent resulted in a rather spectacular cloud of glitter and twigs, serving as a vivid reminder that not all dreams of flight take wing as intended.'), (4, 'Optimized Sled Dynamics', 'This document explores advanced mechanics behind sled propulsion and movement efficiency. Through extensive calculations and enchanted material tests, the elves aim to refine turning capabilities and enhance downhill speed.'), (5, 'Reindeer Harness Design Flaws', 'The latest prototype of our reindeer harness was found to inhibit mobility, causing undue strain on the animals during testing. Observations indicated that the weight distribution was poorly calibrated, necessitating a complete redesign for optimal comfort and performance.'), (6, 'Streamlined Cargo Routing System', 'In our latest endeavor, we have implemented a magical algorithm to optimize the route taken by our toy-laden sleighs. This groundbreaking update minimizes transport time between the workshop and delivery points, ensuring that holiday cheer reaches every home even faster!'), (7, 'Toy Durability Testing Protocols', 'The elves meticulously conducted stress evaluations on the latest toy prototypes to determine their resilience under various conditions. Initial findings indicate that while some designs withstood rigorous play, others required reinforcements to avoid premature wear and tear.'), (8, 'Weather Resistance Breakthrough: Frost Shielding', 'In our recent experiments, we discovered an innovative composite material that effectively repels moisture while withstanding extreme cold. This newfound frost shielding could revolutionize our outdoor toys, ensuring they remain both functional and enchanting, even in the harshest winter conditions.'), (9, 'Safety Compliance Check Overview', 'In the pursuit of enchantment and joy, this document outlines the mandatory safety compliance measures for all workshop operations. Each elf must adhere strictly to these guidelines to ensure the safe transport and handling of our delicate prototypes, thus preventing any unforeseen magical mishaps.'), (10, 'Sleigh Skim Mechanism Upgrade', 'The experiment involved enchanting the underside of the sleigh with a whispering wind charm to achieve unprecedented speeds. Results were alarming, leading to uncontrollable flight trajectories and sudden descents—definitely do not attempt again.'), (11, 'Caution: Enchanted Toy Prototype', 'This design incorporates a reactive magic component that may unpredictably animate in the presence of mischief. Ensure all test environments are secured against spontaneous giggles and potential chaos.'), (12, 'Workshop Experiment Safety Checklist', 'Before embarking on any workshop experiments, ensure all safety goggles are securely fastened to prevent debris from interfered visions. Always double-check that the workspace is free of clutter, as unexpected accidents can arise from even the smallest flurry of trinkets and tools.'), (13, 'Cocoa Bean Roasting Innovations', 'This review explores the latest techniques in roasting cocoa beans to achieve unparalleled flavor profiles. Adjustments to temperature and timing have led to a delightful spectrum of aromas, promising to elevate our confectionery creations to new heights.'), (14, 'Intricate Snowflake Ornament Design', 'This design blueprint outlines the geometric intricacies of a multi-faceted snowflake ornament, emphasizing a balance between elegance and structural integrity. Each arm is meticulously patterned to reflect light, creating a shimmering effect that dances with the seasons, while ensuring optimal symmetry for enchanting visual appeal.'), (15, 'Gift Box Assembly Prototype', 'This prototype outlines the intricate process of assembling the enchanted gift boxes designed to withstand the whims of time and space. Each step must be meticulously executed to ensure that every box not only sparkles with joy but also maintains its magical properties through every unwrapping.'), (16, 'Wrap-It-Up: Innovative Designs', 'This experimental report explores various materials and techniques for creating enchanted wrapping paper that enhances the gift-giving experience. Initial findings suggest that incorporation of shimmering elven dust can amplify the aesthetic appeal while maintaining structural integrity during airborne delivery.'), (17, 'Magical Confection Fusion Results', 'The experimental concoction blended sugar crystals with essence of starlight, resulting in a luminescent treat that sparkles enchantingly. However, a curious side effect was noted: excessive giggling among taste testers, raising questions about potential airborne laughter.'), (18, 'Luminous Ornament Crafting Techniques', 'In our continuous pursuit of radiance, this document outlines innovative methods for creating ornaments that glow with enchantment. Engaging both traditional techniques and modern enchantments, each design is meant to instill joy and sparkle during the festive season.'), (19, 'Magical Energy Conduction Analysis', 'Recent experiments have shown that the flow of magical energy through crystalline conduits behaves unpredictably under varying lunar phases. Further investigation into the correlation between ambient mana levels and energy stability is necessary to optimize enchantment potency.'), (20, 'Elven Workshop Organization Protocols', 'The implementation of open shelving systems has significantly increased accessibility to essential materials, thus enhancing workflow efficiency. Furthermore, the organization of tools into color-coded bins ensures that each elf can swiftly locate their required implements without disrupting the harmony of the workshop.'); ``` We have just one table and a couple of text like columns. That's it, looks like a string searching problem. Let's head to the problem statement! ## Problem > Using the `archive_records` table, search both the `title` and `description` fields for the term "fly". Make sure that you also match for words like "flying", "flight", etc. Boost the results where the term appears in the title and lastly, rank the results by relevance (most relevant first). Provide the elves the top 5 most relevant archived records back. It is a text search use case problem indeed! We need to find and rank the records matching the word `fly`, `flying`, `flight` like those. Kind of tricky if we miss any terms that are not hard-coded. We need to boost the search term in `title`, so there is more weightage if the term appears in title than in description. Makes sense! Let's start simple and move into full-text-search in SQLite! ### Simple String Matching We start by a simple nested `CASE WHEN THEN` condition. We check if the `title` has `fly`, `flying`, `flight`, etc then we set the rank as `2` and add the score from `description` as `1` if the same terms appear in description. So, - If the search term (fly, flight, etc) appears **only** in title, score is `2` - If the search term appears **only** in description, score is `1` - If the search term appears in **both** title and description, the score is `3` since we are adding the scores. - If the search term doesn't appear at all, then the score remains `0`. We simply assign the score based on the appearance of the search term and then order the result based on the computed `rank` and list the top 5. ```sql SELECT id, title, description, ( CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 2 ELSE 0 END + CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_records ORDER BY rank DESC, id ASC LIMIT 5; ``` However, we can limit the search space by eliminating the computation of ranking all the records with a `WHERE` clause since it can optimise only the relevant records before giving out the result set. Hence, we filter only in the cases where the `title` and `description` have the relevant word and then use the condition to order by the `rank` score that we computed and list the top `5` ones. ```sql SELECT id, title, description, ( -- Title matches worth 2 points (boosted relevance) CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 2 ELSE 0 END + -- Description matches worth 1 point CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ORDER BY rank DESC, id ASC LIMIT 5; ``` We can wrap it in a CTE, but that doesn't does anything differently though. With CTE as below, we assign a common score like `1` or `0` based on the `title` or description match. Then in the outer query, we can assign a weight to the rank if the `title` matched as `2` or `description` matched as `1` and add it up to get the same `rank` for that record. ```sql WITH search_results AS ( SELECT id, title, description, CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 1 ELSE 0 END AS title_match, CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END AS desc_match FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ) SELECT id, title, description, (title_match * 2 + desc_match * 1) AS rank FROM search_results ORDER BY rank DESC, id ASC LIMIT 5; ``` ### Assign score based on matching frequency We can also make it better by counting how many times the term appeared in the relevant column and then compute a cumulative rank. This way it becomes more aggressive on the most relevant docs only. So, we write something like this. - Compute the length of the full column example as title with `LENGTH(LOWER(title)` then subtract the length of the word left after removing `fly`, or `flight`, or others. It calculates the difference in length between the original title and the one with 'fly' removed. Each 'fly' is 3 characters, so the difference divided by 3 gives the number of occurrences of 'fly'. This `(LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 ` counts how many times 'fly' appears. Similarly for `flight` and `flying` it is `6`, and so on. - If that appears in title it is multiplied by `2` - If it appears in description, it is kept as is (multiplied by `1`) but you can weight it accordingly, as there could be multiple column to weigh. ```sql SELECT id, title, description, ( ((LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flight', ''))) / 6 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flying', ''))) / 6 ) * 2 + ((LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'fly', ''))) / 3 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flight', ''))) / 6 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flying', ''))) / 6 ) ) AS rank FROM archive_records ORDER BY rank DESC, id ASC LIMIT 5; ``` You can always add a where clause to reduce the search space upfront: ```sql -- Simplified frequency-based ranking SELECT id, title, description, ( ((LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flight', ''))) / 6 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flying', ''))) / 6 ) * 2 + ((LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'fly', ''))) / 3 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flight', ''))) / 6 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flying', ''))) / 6 ) ) AS rank FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ORDER BY rank DESC, id ASC LIMIT 5; ``` This would give us similar results but slightly different as we are assigning rank on the number of times each term appears and not just if it appears once or not. We can even make the list of words and their weight as a CTE and dynamically use it in the actual query: ```sql WITH keywords AS ( SELECT 'fly' AS term, 2.0 AS title_weight, 1.0 AS desc_weight UNION ALL SELECT 'flight', 2.0, 1.0 UNION ALL SELECT 'flying', 2.0, 1.0 ), ranked AS ( SELECT a.id, a.title, a.description, SUM( ((LENGTH(LOWER(a.title)) - LENGTH(REPLACE(LOWER(a.title), k.term, ''))) / LENGTH(k.term)) * k.title_weight + ((LENGTH(LOWER(a.description)) - LENGTH(REPLACE(LOWER(a.description), k.term, ''))) / LENGTH(k.term)) * k.desc_weight ) AS rank FROM archive_records a CROSS JOIN keywords k GROUP BY a.id ) SELECT id, title, description, rank FROM ranked ORDER BY rank DESC, id ASC LIMIT 5; ``` Here we have 2 CTEs now: - `keywords` defining the words to search for and its weight - `ranked` defining the computed sum of weights with frequency on the relevant columns. As you can see, by just adding the keyword in the CTE along with its weight, the rest of the query can work without changing anything. ### Full Text Search We can now also look at the [FTS](https://sqlite.org/fts5.html) or Full Text Search in SQLite. In SQLite, we can create a `VIRTUAL TABLE` which is like - A table computed on the fly - Doesn't exist as physical entity in the database We have [fts5](https://sqlite.org/fts5.html) which we can use to match, equality, or expressions in functions to search for text in columns. ```sql CREATE VIRTUAL TABLE IF NOT EXISTS archive_fts USING fts5( title, description, content=archive_records ); ``` We need to provide the columns that we want to search against. In this case we want to search for `title` and `description` as the column. The `content=archive_records`, links the FTS table to a real table `archive_records` instead of storing its own copy of all text, the FTS table indexes the data from archive_records. Then we want to insert all the records from the `archive_records` for it to make aware of the records existing in the actual table. ```sql INSERT OR IGNORE INTO archive_fts(rowid, title, description) SELECT id, title, description FROM archive_records; ``` Then we can query it like so: ```sql SELECT * FROM archive_fts WHERE archive_fts MATCH 'fly'; ``` We have a couple of option to search against like 1. MATCH 2. Boolean Operator 3. Expressions and Wildcards We simply can filter the records with this ```sql SELECT * FROM archive_fts JOIN archive_records ON archive_fts.rowid = archive_records.id WHERE archive_fts MATCH 'fly OR flight OR flying'; ``` We have added `MATCH 'fly OR flight OR flying'` to only limit the search space on those keywords. ```sql SELECT archive_records.id, archive_records.title, archive_records.description, ( CASE WHEN archive_records.title LIKE '%fly%' OR archive_records.title LIKE '%flight%' OR archive_records.title LIKE '%flying%' THEN 2 ELSE 0 END + CASE WHEN archive_records.description LIKE '%fly%' OR archive_records.description LIKE '%flight%' OR archive_records.description LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_fts JOIN archive_records ON archive_fts.rowid = archive_records.id WHERE archive_fts MATCH 'fly OR flight OR flying' ORDER BY rank DESC, archive_records.id ASC LIMIT 5; ``` This simply will give us the relevant results by doing a fuzzy search which is way better then `LIKE` or `%` wildcard operators in the columns. So, that are the approaches I like taking on day 12. It was fun! Working with full text search for the first time. On to day 13!

Advent of SQL 2025 Day 11: Behavior Score

Fri, 26 Dec 2025 00:00:00 UTC

## Advent of SQL - Day 11, Behavior Score All right, this is day 11 from Advent of SQL. Let's pull in the data. ```sql DROP TABLE IF EXISTS behavior_logs; CREATE TABLE behavior_logs ( id INT PRIMARY KEY, child_id INT, child_name TEXT, behavior_date DATE, score INT ); INSERT INTO behavior_logs (id, child_id, child_name, behavior_date, score) VALUES (1, 1, 'Emma D.', '2025-12-01', 5), (2, 1, 'Emma D.', '2025-12-02', 1), (3, 1, 'Emma D.', '2025-12-03', 3), (4, 1, 'Emma D.', '2025-12-04', 5), (5, 1, 'Emma D.', '2025-12-05', 2), (6, 1, 'Emma D.', '2025-12-06', 2), (7, 1, 'Emma D.', '2025-12-07', 3), (8, 1, 'Emma D.', '2025-12-08', 5), (9, 1, 'Emma D.', '2025-12-09', 4), (10, 1, 'Emma D.', '2025-12-10', 5), (11, 1, 'Emma D.', '2025-12-11', 5), (12, 1, 'Emma D.', '2025-12-12', -1), (13, 1, 'Emma D.', '2025-12-13', 1), (14, 1, 'Emma D.', '2025-12-14', 1), (15, 1, 'Emma D.', '2025-12-15', -1), (16, 1, 'Emma D.', '2025-12-16', 3), (17, 1, 'Emma D.', '2025-12-17', -2), (18, 1, 'Emma D.', '2025-12-18', 1), (19, 1, 'Emma D.', '2025-12-19', 1), (20, 1, 'Emma D.', '2025-12-20', -2), (21, 2, 'Ava X.', '2025-12-01', 0), (22, 2, 'Ava X.', '2025-12-02', -1), (23, 2, 'Ava X.', '2025-12-03', 4), (24, 2, 'Ava X.', '2025-12-04', 0), (25, 2, 'Ava X.', '2025-12-05', 2), (26, 2, 'Ava X.', '2025-12-06', 3), (27, 2, 'Ava X.', '2025-12-07', 5), (28, 2, 'Ava X.', '2025-12-08', 2), (29, 2, 'Ava X.', '2025-12-09', 1), (30, 2, 'Ava X.', '2025-12-10', 5), (31, 2, 'Ava X.', '2025-12-11', 2), (32, 2, 'Ava X.', '2025-12-12', 5), (33, 2, 'Ava X.', '2025-12-13', 5), (34, 2, 'Ava X.', '2025-12-14', 2), (35, 2, 'Ava X.', '2025-12-15', 0), (36, 2, 'Ava X.', '2025-12-16', 0), (37, 2, 'Ava X.', '2025-12-17', 5), (38, 2, 'Ava X.', '2025-12-18', 4), (39, 2, 'Ava X.', '2025-12-19', 5), (40, 2, 'Ava X.', '2025-12-20', 5), (181, 10, 'Ava C.', '2025-12-01', 3), (182, 10, 'Ava C.', '2025-12-02', 0), (183, 10, 'Ava C.', '2025-12-03', 3), (184, 10, 'Ava C.', '2025-12-04', 5), (185, 10, 'Ava C.', '2025-12-05', 5), (186, 10, 'Ava C.', '2025-12-06', 4), (187, 10, 'Ava C.', '2025-12-07', 1), (188, 10, 'Ava C.', '2025-12-08', 4), (189, 10, 'Ava C.', '2025-12-09', 5), (190, 10, 'Ava C.', '2025-12-10', 5), (191, 10, 'Ava C.', '2025-12-11', 5), (192, 10, 'Ava C.', '2025-12-12', 0), (193, 10, 'Ava C.', '2025-12-13', 0), (194, 10, 'Ava C.', '2025-12-14', 3), (195, 10, 'Ava C.', '2025-12-15', 1), (196, 10, 'Ava C.', '2025-12-16', 3), (197, 10, 'Ava C.', '2025-12-17', -1), (198, 10, 'Ava C.', '2025-12-18', 0), (199, 10, 'Ava C.', '2025-12-19', 5), (200, 10, 'Ava C.', '2025-12-20', 4), (241, 13, 'Ava R.', '2025-12-01', 3), (242, 13, 'Ava R.', '2025-12-02', 2), (243, 13, 'Ava R.', '2025-12-03', 2), (244, 13, 'Ava R.', '2025-12-04', 1), (245, 13, 'Ava R.', '2025-12-05', -1), (246, 13, 'Ava R.', '2025-12-06', -1), (247, 13, 'Ava R.', '2025-12-07', 2), (248, 13, 'Ava R.', '2025-12-08', 5), (249, 13, 'Ava R.', '2025-12-09', 0), (250, 13, 'Ava R.', '2025-12-10', 5), (251, 13, 'Ava R.', '2025-12-11', 2), (252, 13, 'Ava R.', '2025-12-12', -1), (253, 13, 'Ava R.', '2025-12-13', 2), (254, 13, 'Ava R.', '2025-12-14', 3), (255, 13, 'Ava R.', '2025-12-15', 2), (256, 13, 'Ava R.', '2025-12-16', -1), (257, 13, 'Ava R.', '2025-12-17', -2), (258, 13, 'Ava R.', '2025-12-18', -4), (259, 13, 'Ava R.', '2025-12-19', -3), (260, 13, 'Ava R.', '2025-12-20', 2), (1961, 99, 'Ava X.', '2025-12-01', 2), (1962, 99, 'Ava X.', '2025-12-02', -2), (1963, 99, 'Ava X.', '2025-12-03', -1), (1964, 99, 'Ava X.', '2025-12-04', -2), (1965, 99, 'Ava X.', '2025-12-05', 3), (1966, 99, 'Ava X.', '2025-12-06', -1), (1967, 99, 'Ava X.', '2025-12-07', 0), (1968, 99, 'Ava X.', '2025-12-08', 1), (1969, 99, 'Ava X.', '2025-12-09', 0), (1970, 99, 'Ava X.', '2025-12-10', 0), (1971, 99, 'Ava X.', '2025-12-11', 3), (1972, 99, 'Ava X.', '2025-12-12', 4), (1973, 99, 'Ava X.', '2025-12-13', 4), (1974, 99, 'Ava X.', '2025-12-14', 0), (1975, 99, 'Ava X.', '2025-12-15', 3), (1976, 99, 'Ava X.', '2025-12-16', -1), (1977, 99, 'Ava X.', '2025-12-17', -1), (1978, 99, 'Ava X.', '2025-12-18', 3), (1979, 99, 'Ava X.', '2025-12-19', 3), (1980, 99, 'Ava X.', '2025-12-20', -3); ``` No hiccups! Good to go. We just have one table today. ```sql SELECT * FROM behavior_logs; ``` ``` sqlite> .read day11-inserts.sql sqlite> .mode table sqlite> .schema CREATE TABLE behavior_logs ( id INT PRIMARY KEY, child_id INT, child_name TEXT, behavior_date DATE, score INT ); sqlite> SELECT * FROM behavior_logs LIMIT 20; +----+----------+------------+---------------+-------+ | id | child_id | child_name | behavior_date | score | +----+----------+------------+---------------+-------+ | 1 | 1 | Emma D. | 2025-12-01 | 5 | | 2 | 1 | Emma D. | 2025-12-02 | 1 | | 3 | 1 | Emma D. | 2025-12-03 | 3 | | 4 | 1 | Emma D. | 2025-12-04 | 5 | | 5 | 1 | Emma D. | 2025-12-05 | 2 | | 6 | 1 | Emma D. | 2025-12-06 | 2 | | 7 | 1 | Emma D. | 2025-12-07 | 3 | | 8 | 1 | Emma D. | 2025-12-08 | 5 | | 9 | 1 | Emma D. | 2025-12-09 | 4 | | 10 | 1 | Emma D. | 2025-12-10 | 5 | | 11 | 1 | Emma D. | 2025-12-11 | 5 | | 12 | 1 | Emma D. | 2025-12-12 | -1 | | 13 | 1 | Emma D. | 2025-12-13 | 1 | | 14 | 1 | Emma D. | 2025-12-14 | 1 | | 15 | 1 | Emma D. | 2025-12-15 | -1 | | 16 | 1 | Emma D. | 2025-12-16 | 3 | | 17 | 1 | Emma D. | 2025-12-17 | -2 | | 18 | 1 | Emma D. | 2025-12-18 | 1 | | 19 | 1 | Emma D. | 2025-12-19 | 1 | | 20 | 1 | Emma D. | 2025-12-20 | -2 | +----+----------+------------+---------------+-------+ sqlite> SELECT * FROM behavior_logs WHERE child_name LIKE 'Ava'; sqlite> SELECT * FROM behavior_logs WHERE child_name LIKE 'Ava%'; +------+----------+------------+---------------+-------+ | id | child_id | child_name | behavior_date | score | +------+----------+------------+---------------+-------+ | 21 | 2 | Ava X. | 2025-12-01 | 0 | | 22 | 2 | Ava X. | 2025-12-02 | -1 | | 23 | 2 | Ava X. | 2025-12-03 | 4 | | 24 | 2 | Ava X. | 2025-12-04 | 0 | | 25 | 2 | Ava X. | 2025-12-05 | 2 | | 26 | 2 | Ava X. | 2025-12-06 | 3 | | 27 | 2 | Ava X. | 2025-12-07 | 5 | | 28 | 2 | Ava X. | 2025-12-08 | 2 | | 29 | 2 | Ava X. | 2025-12-09 | 1 | | 30 | 2 | Ava X. | 2025-12-10 | 5 | | 31 | 2 | Ava X. | 2025-12-11 | 2 | | 32 | 2 | Ava X. | 2025-12-12 | 5 | | 33 | 2 | Ava X. | 2025-12-13 | 5 | | 34 | 2 | Ava X. | 2025-12-14 | 2 | | 35 | 2 | Ava X. | 2025-12-15 | 0 | | 36 | 2 | Ava X. | 2025-12-16 | 0 | | 37 | 2 | Ava X. | 2025-12-17 | 5 | | 38 | 2 | Ava X. | 2025-12-18 | 4 | | 39 | 2 | Ava X. | 2025-12-19 | 5 | | 40 | 2 | Ava X. | 2025-12-20 | 5 | | 181 | 10 | Ava C. | 2025-12-01 | 3 | | 182 | 10 | Ava C. | 2025-12-02 | 0 | | 183 | 10 | Ava C. | 2025-12-03 | 3 | | 184 | 10 | Ava C. | 2025-12-04 | 5 | | 185 | 10 | Ava C. | 2025-12-05 | 5 | | 186 | 10 | Ava C. | 2025-12-06 | 4 | | 187 | 10 | Ava C. | 2025-12-07 | 1 | | 188 | 10 | Ava C. | 2025-12-08 | 4 | | 189 | 10 | Ava C. | 2025-12-09 | 5 | | 190 | 10 | Ava C. | 2025-12-10 | 5 | | 191 | 10 | Ava C. | 2025-12-11 | 5 | | 192 | 10 | Ava C. | 2025-12-12 | 0 | | 193 | 10 | Ava C. | 2025-12-13 | 0 | | 194 | 10 | Ava C. | 2025-12-14 | 3 | | 195 | 10 | Ava C. | 2025-12-15 | 1 | | 196 | 10 | Ava C. | 2025-12-16 | 3 | | 197 | 10 | Ava C. | 2025-12-17 | -1 | | 198 | 10 | Ava C. | 2025-12-18 | 0 | | 199 | 10 | Ava C. | 2025-12-19 | 5 | | 200 | 10 | Ava C. | 2025-12-20 | 4 | | 241 | 13 | Ava R. | 2025-12-01 | 3 | | 242 | 13 | Ava R. | 2025-12-02 | 2 | | 243 | 13 | Ava R. | 2025-12-03 | 2 | | 244 | 13 | Ava R. | 2025-12-04 | 1 | | 245 | 13 | Ava R. | 2025-12-05 | -1 | | 246 | 13 | Ava R. | 2025-12-06 | -1 | | 247 | 13 | Ava R. | 2025-12-07 | 2 | | 248 | 13 | Ava R. | 2025-12-08 | 5 | | 249 | 13 | Ava R. | 2025-12-09 | 0 | | 250 | 13 | Ava R. | 2025-12-10 | 5 | | 251 | 13 | Ava R. | 2025-12-11 | 2 | | 252 | 13 | Ava R. | 2025-12-12 | -1 | | 253 | 13 | Ava R. | 2025-12-13 | 2 | | 254 | 13 | Ava R. | 2025-12-14 | 3 | | 255 | 13 | Ava R. | 2025-12-15 | 2 | | 256 | 13 | Ava R. | 2025-12-16 | -1 | | 257 | 13 | Ava R. | 2025-12-17 | -2 | | 258 | 13 | Ava R. | 2025-12-18 | -4 | | 259 | 13 | Ava R. | 2025-12-19 | -3 | | 260 | 13 | Ava R. | 2025-12-20 | 2 | | 1961 | 99 | Ava X. | 2025-12-01 | 2 | | 1962 | 99 | Ava X. | 2025-12-02 | -2 | | 1963 | 99 | Ava X. | 2025-12-03 | -1 | | 1964 | 99 | Ava X. | 2025-12-04 | -2 | | 1965 | 99 | Ava X. | 2025-12-05 | 3 | | 1966 | 99 | Ava X. | 2025-12-06 | -1 | | 1967 | 99 | Ava X. | 2025-12-07 | 0 | | 1968 | 99 | Ava X. | 2025-12-08 | 1 | | 1969 | 99 | Ava X. | 2025-12-09 | 0 | | 1970 | 99 | Ava X. | 2025-12-10 | 0 | | 1971 | 99 | Ava X. | 2025-12-11 | 3 | | 1972 | 99 | Ava X. | 2025-12-12 | 4 | | 1973 | 99 | Ava X. | 2025-12-13 | 4 | | 1974 | 99 | Ava X. | 2025-12-14 | 0 | | 1975 | 99 | Ava X. | 2025-12-15 | 3 | | 1976 | 99 | Ava X. | 2025-12-16 | -1 | | 1977 | 99 | Ava X. | 2025-12-17 | -1 | | 1978 | 99 | Ava X. | 2025-12-18 | 3 | | 1979 | 99 | Ava X. | 2025-12-19 | 3 | | 1980 | 99 | Ava X. | 2025-12-20 | -3 | +------+----------+------------+---------------+-------+ sqlite> ``` Let's get to the problem of day 11 ## Problem > Calculate the 7-day rolling average behavior score for each child. Identify any child whose rolling average drops below 0. For those children with a rolling average below 0, return the `child_id`, `child_name`, `behavior_date` (this will be the latest date in the 7-day rolling average), and the calculated 7-day rolling average. Only include results with a `behavior_date` of `December 7, 2025` or later, ensuring that each rolling average is based on a full 7 days of data. > > Order the results by `behavior_date` and then `child_name`. So, we need to do what? - Group by child_id - Compute the rolling average for the past 7 days (so if there is a data for 20 days from 1st to 20th December, we'll only consider the average from 14th to 20th December) - Order by behavior_date and child_name. ### Using Simple Join We have to first grab the rolling average which is only for the past 7 days per child. So, to do that, we can self join the `behavior_log` table on the condition that the behavior date (from right table) is between the current behavior date (from left table) and 6 days past that for each child. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id; ``` We basically joined the table `behavior_logs` with itself, i.e. the right and left tables are the same. So, we join on the condition of same child_id, and then look for dates between the past 6 days and the current behavior date (we included the 7th day in the behavior date, so hence subtracting 6 days from that date). We group by `child_id`, so that we get a single row for each `child_id`. We compute the `AVG(score)` to get the average score for each day in the range of the past 7 days. You see the issue, we need to compute the average for each week, and not just the last, it will give us the last most score for a child. We need to also group by the behavior date so as to keep week unique and have multiple entries for a child. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date; ``` Now, we have grouped by `behavior_date` so that each child can have multiple entries for the weeks that we have logs for. Now we can further filter it. We also need to filter when the `rolling_average` is less than 0, that is the child had a bad week overall. So that will be in a `HAVING` condition and not a `WHERE` condition since `AVG` is a aggregate function, we can't reference the `rolling_average` in the where clause, it won't be defined there yet. So, we use `HAVING` to filter `rolling_average` as less than `0`. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0; ``` We are now down to the rows that only have `bad` weeks for the child in various weeks. There is one catch however, we can't use week periods before `7th December` because we don't have enough data to compute the rolling average for the 7 day score. Hence we only include the records having logs after `7th December`. This again would come in the `HAVING` clause as we are deciding the final behavior date from the week and not the individual logs from the table. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0 AND behavior_logs.behavior_date >= '2025-12-07' ``` Now, the final piece is the order. We need to order it by the `behavior_date` and the `child_name` as requested. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0 AND behavior_logs.behavior_date >= '2025-12-07' ORDER BY behavior_logs.behavior_date, behavior_logs.child_name; ``` This is a simple solution. Easy to understand, and explain, not quite short and crisp. We can even do this in a sub query without a JOIN like so ### Using Sub-query We take the conditions from the `JOIN` and replace that in a sub-query like so. ```sql SELECT current_logs.child_id, current_logs.child_name, current_logs.behavior_date, ( SELECT AVG(past_logs.score) FROM behavior_logs past_logs WHERE past_logs.child_id = current_logs.child_id AND past_logs.behavior_date BETWEEN date(current_logs.behavior_date, '-6 days') AND current_logs.behavior_date ) AS rolling_avg FROM behavior_logs current_logs WHERE current_logs.behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY current_logs.behavior_date, current_logs.child_name; ``` Since its a subquery, we don't need to group by or add a having clause to filter the rolling average and the behavior date. ### Using Sub-query and Window Function We can take the above query and instead of join, we can write a sub-query to compute the rolling average using window function. ```sql SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ``` We use the `AVG(score) OVER ()` this is a partition. We partition or create a window for each child and we order by the behavior date and then create a sliding window for the past 7 days. The `ROWS BETWEEN 6 PRECEDING AND CURRENT ROW` defines a 7-row sliding window for each child. Every row gets its own window, and each window is separate per child because of PARTITION BY `child_id`. So we get a full per day rolling average for each child with this query. This had around 2400 rows. Now, we need filter it down to the only rows where the rolling average is less than `0` and we don't include the average for days before the `7th December` as we don't have enough days before that to compute the 7-day rolling average. But this is a query in itself we can't reference the `rolling_average` in the where clause as its not available there. We haven't grouped by anything explicitly so we can't use `HAVING`, we need to wrap it in a sub query to get the rolling average as well as the behavior date condition. ```sql SELECT child_id, child_name, behavior_date, rolling_avg FROM ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0; ``` This will filter down the rows. Now, we also need to order by the `behavior_date` and then `child_name`. ```sql SELECT child_id, child_name, behavior_date, rolling_avg FROM ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY behavior_date, child_name; ``` So, that is again the same stuff just with window function. We can even do this with that subquery wrapped in a CTE (it just looks and reads good, nothing different really, other than you can use the CTE in the same query multiple times, here we don't need to do it) ### Using CTE and Window Function We just take the above sub-query and wrap it in a CTE. ```sql WITH rolling AS ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) SELECT child_id, child_name, behavior_date, rolling_avg FROM rolling WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY behavior_date, child_name; ``` The rest remains the same, we just reference the `rolling` as the CTE and grab the necessary details in the query using that CTE. That should be it for the Day 11! Some cool CTE, Window function and pratical use case for computing rolling averages, loved it! On to the day 12!

Advent of SQL 2025 Day 10: Misdelivered Presents

Thu, 25 Dec 2025 00:00:00 UTC

## Advent of SQL, Day 10 - Misdelivered Presents It's already day 10? We just need 5 more days now! Whoa! that flew by swiftly. Let's pull in the data. This is the SQL for day 10 in SQLite. ```sql DROP TABLE IF EXISTS misdelivered_presents; DROP TABLE IF EXISTS deliveries; CREATE TABLE deliveries ( id INT PRIMARY KEY, child_name TEXT, delivery_location TEXT, gift_name TEXT, scheduled_at TIMESTAMP ); CREATE TABLE misdelivered_presents ( id INT PRIMARY KEY, child_name TEXT, delivery_location TEXT, gift_name TEXT, scheduled_at TIMESTAMP, flagged_at TIMESTAMP, reason TEXT ); INSERT INTO deliveries (id, child_name, delivery_location, gift_name, scheduled_at) VALUES (1, 'Omar Q.', '45 Maple Street', 'storybook collection', '2025-12-24 21:09:00'), (2, 'Sofia K.', '77 Snowflake Road', 'plush reindeer', '2025-12-24 18:35:00'), (3, 'Mila N.', 'The Vibes', 'storybook collection', '2025-12-24 21:09:00'), (4, 'Elias M.', 'Frost Hollow Cabin', 'board game', '2025-12-24 20:31:00'), (5, 'Ravi P.', '45 Maple Street', 'wooden train set', '2025-12-24 18:23:00'), (6, 'Jonah W.', '77 Snowflake Road', 'plush reindeer', '2025-12-24 20:34:00'), (7, 'Ava J.', '123 Evergreen Lane', 'board game', '2025-12-24 21:03:00'), (8, 'Omar Q.', '77 Snowflake Road', 'board game', '2025-12-24 18:56:00'), (9, 'Nia G.', 'Frost Hollow Cabin', 'teddy bear', '2025-12-24 21:27:00'), (10, 'Zara S.', 'North Pole Annex', 'wooden train set', '2025-12-24 20:58:00'), (11, 'Ravi P.', 'Frost Hollow Cabin', 'puzzle box', '2025-12-24 18:39:00'), (12, 'Jonah W.', '123 Evergreen Lane', 'puzzle box', '2025-12-24 18:23:00'), (13, 'Ravi P.', 'North Pole Annex', 'storybook collection', '2025-12-24 21:36:00'), (14, 'Lena F.', 'North Pole Annex', 'teddy bear', '2025-12-24 21:26:00'), (15, 'Ava J.', 'North Pole Annex', 'snow globe', '2025-12-24 18:31:00'), (16, 'Elias M.', 'Frost Hollow Cabin', 'robot toy', '2025-12-24 20:21:00'), (17, 'Sofia K.', 'Frost Hollow Cabin', 'teddy bear', '2025-12-24 20:27:00'), (18, 'Jonah W.', '77 Snowflake Road', 'storybook collection', '2025-12-24 20:49:00'), (19, 'Jonah W.', 'Frost Hollow Cabin', 'art supplies', '2025-12-24 21:38:00'), (20, 'Jonah W.', '123 Evergreen Lane', 'storybook collection', '2025-12-24 19:11:00'); INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason) VALUES (601, 'Priya D.', 'The Vibes', 'plush reindeer', '2025-12-24 14:00:00', '2025-12-24 14:05:00', 'Invalid delivery location'), (602, 'Lena F.', 'Abandoned Lighthouse', 'board game', '2025-12-22 06:00:00', '2025-12-22 06:05:00', 'Invalid delivery location'), (603, 'Caleb O.', 'Drifting Igloo', 'board game', '2025-12-24 06:00:00', '2025-12-24 06:05:00', 'Invalid delivery location'), (604, 'Mateo C.', 'The Vibes', 'art supplies', '2025-12-22 04:00:00', '2025-12-22 04:05:00', 'Invalid delivery location'), (605, 'Hiro T.', 'The Vibes', 'robot toy', '2025-12-24 08:00:00', '2025-12-24 08:05:00', 'Invalid delivery location'), (606, 'Priya D.', 'Volcano Rim', 'puzzle box', '2025-12-22 08:00:00', '2025-12-22 08:05:00', 'Invalid delivery location'), (607, 'Nia G.', 'Abandoned Lighthouse', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (608, 'Elias M.', 'Drifting Igloo', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (609, 'Ravi P.', 'Volcano Rim', 'board game', '2025-12-24 02:00:00', '2025-12-24 02:05:00', 'Invalid delivery location'), (610, 'Hiro T.', 'Abandoned Lighthouse', 'science kit', '2025-12-23 20:00:00', '2025-12-23 20:05:00', 'Invalid delivery location'), (611, 'Priya D.', 'Drifting Igloo', 'puzzle box', '2025-12-22 21:00:00', '2025-12-22 21:05:00', 'Invalid delivery location'), (612, 'Hiro T.', 'Volcano Rim', 'art supplies', '2025-12-23 09:00:00', '2025-12-23 09:05:00', 'Invalid delivery location'), (613, 'Jonah W.', 'Abandoned Lighthouse', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (614, 'Omar Q.', 'Volcano Rim', 'art supplies', '2025-12-22 01:00:00', '2025-12-22 01:05:00', 'Invalid delivery location'), (615, 'Omar Q.', 'Drifting Igloo', 'science kit', '2025-12-23 20:00:00', '2025-12-23 20:05:00', 'Invalid delivery location'), (616, 'Omar Q.', 'Abandoned Lighthouse', 'teddy bear', '2025-12-24 12:00:00', '2025-12-24 12:05:00', 'Invalid delivery location'), (617, 'Zara S.', 'Volcano Rim', 'wooden train set', '2025-12-24 12:00:00', '2025-12-24 12:05:00', 'Invalid delivery location'), (618, 'Omar Q.', 'Abandoned Lighthouse', 'teddy bear', '2025-12-23 15:00:00', '2025-12-23 15:05:00', 'Invalid delivery location'), (619, 'Caleb O.', 'The Vibes', 'teddy bear', '2025-12-24 14:00:00', '2025-12-24 14:05:00', 'Invalid delivery location'), (620, 'Nia G.', 'Abandoned Lighthouse', 'board game', '2025-12-23 03:00:00', '2025-12-23 03:05:00', 'Invalid delivery location'); ``` ```sql SELECT * FROM deliveries; SELECT * FROM misdelivered_presents; ``` We have two tables. Almost the same with critical logical distinction among them and one extra column. Let's see the problem to check it. ## Problem > Clean-up the deliveries table to remove any records where the delivery_location is 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'. > > Move those records to the misdelivered_presents with all the same columns as deliveries plus a flagged_at column with the current time and a reason column with "Invalid delivery location" listed as the reason for each moved record. > > Make sure your final step shows the misdelivered_presents records that you just moved (i.e. don't include any existing records from the misdelivered_presents table). Ok, this looks some easy problem. - SELECT some data - INSERT that data into other table - DELETE that data from the original table - SELECT the newly inserted data in the other table Right? Unless! > Santa turned to you. > > “I don’t want this done in five steps,” he said. “And I don’t want any re-selecting. Move the problem presents out of the delivery system, log them in the vault, and show me exactly what you moved.” Ouch Santa! Don't be lazy! Be smart he says! Huhh! Ok, at least let's check both the tables, how many rows they have, and the rows that we want to move around. ```sql SELECT COUNT(*) AS delivery_count FROM deliveries; SELECT COUNT(*) AS misdelivered_present_count FROM misdelivered_presents; SELECT COUNT(*) AS misdelivered_deliveries_count FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ); SELECT COUNT(*) AS misdelivered_present_count FROM misdelivered_presents WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ); ``` ``` sqlite> SELECT COUNT(*) FROM deliveries; +----------+ | COUNT(*) | +----------+ | 600 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents; +----------+ | COUNT(*) | +----------+ | 50 | +----------+ sqlite> SELECT COUNT(*) FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); +----------+ | COUNT(*) | +----------+ | 103 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); +----------+ | COUNT(*) | +----------+ | 50 | +----------+ sqlite> ``` OK, so we need to delete the `103` records from the `deliveries` table and move them into the `misdelivered_presents` at the same time. How are we going to do that in SQLite. We can try - DELETE FROM deliveries with a SELECT from deliveries - Then INSERT the deleted data into misdelivered_presents Could that work? Let's see ```sql WITH moved AS ( DELETE FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ) RETURNING id, child_name, delivery_location, gift_name, scheduled_at, datetime('now') AS flagged_at, 'Invalid delivery location' AS reason ) SELECT * FROM moved; ``` ``` ``` Ops! Can't generate a CTE with delete in it. That's nasty. Thought that could shove it in the insert into the misdelivered table. But it should be other way then? Insert first and then use the data to delete? ```sql WITH inserted_data AS ( INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at) SELECT id, child_name, delivery_location, gift_name, scheduled_at FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); ``` ``` sqlite> WITH inserted_data AS ( INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at) SELECT id, child_name, delivery_location, gift_name, scheduled_at FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); Parse error: near "INSERT": syntax error rt the selected rows into misdelivered_presents INSERT INTO misdelivered_pre error here ---^ sqlite> ``` Tried it, didn't work Quoting the documentation here. > Common Table Expressions or CTEs act like temporary views that exist only for the duration of a single SQL statement. There are two kinds of common table expressions: "ordinary" and "recursive". Ordinary common table expressions are helpful for making queries easier to understand by factoring subqueries out of the main SQL statement. Recursive common table expressions provide the ability to do hierarchical or recursive queries of trees and graphs, a capability that is not otherwise available in the SQL language. I think it doesn't support CTEs with delete or insert! Sigh! Now? BEGIN COMMIT? Atomic Transactions? Yeah! Santa wanted it in one go right? That's not possible in SQLite but at least everything will happen or nothing will. ```sql BEGIN; WITH misdelivered_deliveries AS ( SELECT * FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) SELECT id, child_name, delivery_location, gift_name, scheduled_at, DATETIME('now'), 'Invalid delivery location' FROM misdelivered_deliveries RETURNING *; DELETE FROM deliveries WHERE id IN ( SELECT id FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ); COMMIT; ``` So here, we defined the CTE that selects data from `deliveries` and uses it to insert into the `misdelivered_deliveries` table. Then a separate query to delete that data from the `deliveries` table. Yeah! I mean I don't think there could be other way to it! We might or can use triggers to insert into one table when inserted in one table. But I think that is too much of a farfetched solution. We might create a trigger and instantly delete it as it could populate unwanted data if kept in the database. ### Trigger to insert when deleted We can create a `TRIGGER` to insert into `misdelivered_presents` when something is deleted from `deliveries` table. We will separately have to delete the records from the `deliveries` table. But the insert will happen automatically after the deletion. Opening a fresh instance of the database! ```sql CREATE TRIGGER move_misdelivered_presents BEFORE DELETE ON deliveries FOR EACH ROW WHEN OLD.delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') BEGIN INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) VALUES ( OLD.id, OLD.child_name, OLD.delivery_location, OLD.gift_name, OLD.scheduled_at, DATETIME('now'), 'Invalid delivery location' ); END; ``` This will create the trigger to insert the row into `misdelivered_presents` table when deleted from the `deliveries` table. ```sql DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') RETURNING *; ``` Ops! We are returning from the DELETE statement which is wrong as the problem stated to select from the `misdelivered_presents` table. > Make sure your final step shows the misdelivered_presents records that you just moved (i.e. don't include any existing records from the misdelivered_presents table). This is invalid then! Though its not technically atomic. It happens before the delete, so it can mess up things. ``` sqlite> .read day10-inserts.sql sqlite> .mode table sqlite> CREATE TRIGGER move_misdelivered_presents BEFORE DELETE ON deliveries FOR EACH ROW WHEN OLD.delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') BEGIN INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) VALUES ( OLD.id, OLD.child_name, OLD.delivery_location, OLD.gift_name, OLD.scheduled_at, DATETIME('now'), 'Invalid delivery location' ); END; sqlite> DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') RETURNING *; +-----+------------+----------------------+----------------------+---------------------+ | id | child_name | delivery_location | gift_name | scheduled_at | +-----+------------+----------------------+----------------------+---------------------+ | 3 | Mila N. | The Vibes | storybook collection | 2025-12-24 21:09:00 | | 22 | Lena F. | Abandoned Lighthouse | plush reindeer | 2025-12-24 19:08:00 | | 23 | Mila N. | Abandoned Lighthouse | storybook collection | 2025-12-24 20:42:00 | | 29 | Mateo C. | Volcano Rim | plush reindeer | 2025-12-24 21:44:00 | | 31 | Nia G. | Drifting Igloo | robot toy | 2025-12-24 19:57:00 | ... ... | 582 | Zara S. | The Vibes | teddy bear | 2025-12-24 21:20:00 | | 585 | Layla B. | Abandoned Lighthouse | wooden train set | 2025-12-24 18:39:00 | | 587 | Nia G. | Volcano Rim | storybook collection | 2025-12-24 18:35:00 | | 596 | Omar Q. | Abandoned Lighthouse | puzzle box | 2025-12-24 19:28:00 | +-----+------------+----------------------+----------------------+---------------------+ sqlite> DROP TRIGGER move_misdelivered_presents; sqlite> SELECT COUNT(*) FROM deliveries; +----------+ | COUNT(*) | +----------+ | 497 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents; +----------+ | COUNT(*) | +----------+ | 153 | +----------+ sqlite> ``` So, it gets the result but I don't think the `TRIGGER` is a solution to this. In SQLite, Atomic transaction using begin and end is the only thing to go for right? Some one prove Santa that it can't be done in SQLite in one query? Please! That's it from day 10, I have spent enough time on this, banging my head on the sqlite shell! Off to day 11 tomorrow!

Advent of SQL 2025 Day 9: Evergreen Market Orders

Wed, 24 Dec 2025 00:00:00 UTC

## Advent of SQL, Day 9 - Evergreen Market Orders We are on day 9 of advent of SQL, and I feel good so far. Let's see what we learn today? Let's get the inserts for the day. ``` sqlite> .read day9-inserts.sql sqlite> .schema CREATE TABLE orders ( id INT PRIMARY KEY, customer_id INT, created_at TIMESTAMP, order_data JSONB ); sqlite> .mode table sqlite> SELECT * FROM orders limit 10; +----+-------------+---------------------+--------------------------------------------------------------+ | id | customer_id | created_at | order_data | +----+-------------+---------------------+--------------------------------------------------------------+ | 1 | 1 | 2025-11-21 13:08:22 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | | | | | e}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 2 | 1 | 2025-11-21 18:42:58 | {"shipping": {"method": "overnight"}, "risk": {"flag": "high | | | | | "}, "gift": {"wrapped": false}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 3 | 1 | 2025-11-21 21:01:46 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | | | | | se}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 4 | 1 | 2025-11-24 13:17:27 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | | | | | e}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 5 | 1 | 2025-11-24 21:09:46 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": fa | | | | | lse}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 6 | 1 | 2025-11-25 07:24:55 | {"shipping": {"method": "standard"}, "risk": {"flag": "mediu | | | | | m"}, "gift": {"wrapped": true}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 7 | 1 | 2025-11-25 17:42:36 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | | | | | se}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 8 | 1 | 2025-11-27 02:34:24 | {"shipping": {"method": "express"}, "gift": {"wrapped": true | | | | | }} | +----+-------------+---------------------+--------------------------------------------------------------+ | 9 | 1 | 2025-11-30 22:43:54 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": tr | | | | | ue}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 10 | 1 | 2025-12-01 04:03:33 | {"shipping": {"method": "express"}, "risk": {"flag": "medium | | | | | "}, "gift": {"wrapped": false}} | +----+-------------+---------------------+--------------------------------------------------------------+ sqlite> ``` Looks like we will deal with JSON today, seems exciting. I haven't dealt with JSON in SQLite yet, today will break it. Let's get some sample inserts for you to play in the browser. LIMITING TO 20, there are more than 400 rows! ```sql DROP TABLE IF EXISTS orders; CREATE TABLE orders ( id INT PRIMARY KEY, customer_id INT, created_at TIMESTAMP, order_data JSONB ); INSERT INTO orders (id, customer_id, created_at, order_data) VALUES (1, 1, '2025-11-21 13:08:22', '{"shipping": {"method": "standard"}, "gift": {"wrapped": true}}'), (2, 1, '2025-11-21 18:42:58', '{"shipping": {"method": "overnight"}, "risk": {"flag": "high"}, "gift": {"wrapped": false}}'), (3, 1, '2025-11-21 21:01:46', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (4, 1, '2025-11-24 13:17:27', '{"shipping": {"method": "standard"}, "gift": {"wrapped": true}}'), (5, 1, '2025-11-24 21:09:46', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": false}}'), (6, 1, '2025-11-25 07:24:55', '{"shipping": {"method": "standard"}, "risk": {"flag": "medium"}, "gift": {"wrapped": true}}'), (7, 1, '2025-11-25 17:42:36', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (8, 1, '2025-11-27 02:34:24', '{"shipping": {"method": "express"}, "gift": {"wrapped": true}}'), (9, 1, '2025-11-30 22:43:54', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": true}}'), (10, 1, '2025-12-01 04:03:33', '{"shipping": {"method": "express"}, "risk": {"flag": "medium"}, "gift": {"wrapped": false}}'), (11, 1, '2025-12-02 05:19:10', '{"shipping": {"method": "overnight"}, "risk": {"flag": "low"}, "gift": {"wrapped": true}}'), (12, 1, '2025-12-03 16:25:56', '{"shipping": {"method": "overnight"}, "risk": {"flag": "medium"}, "gift": {"wrapped": true}}'), (13, 1, '2025-12-10 19:34:28', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (14, 1, '2025-12-16 19:23:53', '{"shipping": {"method": "express"}, "gift": {"wrapped": false}}'), (15, 2, '2025-11-23 19:11:23', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": true}}'), (16, 2, '2025-11-28 15:23:27', '{"shipping": {"method": "express"}, "gift": {"wrapped": true}}'), (17, 2, '2025-11-30 12:05:36', '{"shipping": {"method": "overnight"}, "risk": {"flag": "low"}, "gift": {"wrapped": true}}'), (18, 2, '2025-12-03 07:03:06', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (19, 2, '2025-12-07 13:55:13', '{"shipping": {"method": "express"}, "risk": {"flag": "high"}, "gift": {"wrapped": false}}'), (20, 2, '2025-12-08 07:17:31', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'); ``` ```sql SELECT * FROM orders; ``` Let's get to the problem now. ## Problem > Build a report using the orders table that shows the latest order for each customer, along with their requested shipping method, gift wrap choice (as true or false), and the risk flag in separate columns. > > Order the report by the most recent order first so Evergreen Market can reach out to them ASAP. Ok, so we need for each customer the latest order with the following details: - shipping method - gift wrap choice - risk flag These all I think are in the same column as a JSON string or blob. We need to extract those out from the column. Let's first check the `orders` table. It has a few columns: - id - customer_id - created_at - order_data We do require the `order_data` as that is the column that has JSON. Also the problem said to give the most recent order, so we need to order by `created_at` in a reverse way, the latest first. Also we need it per customer, so we need to group by `customer_id`. Let's see how to get the data inside JSON in SQLite. ### JSON Extract Well we have [json_extract](https://sqlite.org/json1.html#jex) which can give us the value of the key from the given json data string. The function takes in the column name from the table which would be the column containing the json data and then the second parameter is the path to the key, in this case if we want to get the `method` from the `shipping` key, we can use `$.shipping.method` which means from the root `$` get the `shipping` key, and inside that (`shipping` key) give the value inside the `method` key. If the path is not present, in our case the `risk` key is very rarely present in the actual original json data, the function skips the further key lookup, it returns `NULL`. ```sql SELECT json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; ``` ``` sqlite> select json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; +------------------------------------------------------+ | json_extract(orders.order_data, '$.shipping.method') | +------------------------------------------------------+ | standard | | overnight | | standard | | standard | | overnight | +------------------------------------------------------+ sqlite> select *, json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | id | customer_id | created_at | order_data | json_extract(orders.order_data, '$.shipping.method') | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 1 | 1 | 2025-11-21 13:08:22 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | standard | | | | | e}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 2 | 1 | 2025-11-21 18:42:58 | {"shipping": {"method": "overnight"}, "risk": {"flag": "high | overnight | | | | | "}, "gift": {"wrapped": false}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 3 | 1 | 2025-11-21 21:01:46 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | standard | | | | | se}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 4 | 1 | 2025-11-24 13:17:27 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | standard | | | | | e}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 5 | 1 | 2025-11-24 21:09:46 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": fa | overnight | | | | | lse}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ sqlite> ``` Let's grab the other two. ```sql SELECT json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders; ``` ``` sqlite> select json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders LIMIT 5; +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ | json_extract(order_data, '$.shipping.method') | json_extract(order_data, '$.gift.wrapped') | json_extract(order_data, '$.risk.flag') | +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ | standard | 1 | | | overnight | 0 | high | | standard | 0 | | | standard | 1 | | | overnight | 0 | | +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ sqlite> ``` Hmm, interesting, it automatically converted `false` to `0`, and `true` to `1` as you can see in the actual json data it was `true` and `false`. Nice, but could go the other way as well, if you were checking for true or false instead of 0 and 1. Both are same in SQLite, they are treated as truth and false values. We have all the things we need, now? Well, let's group by the `customer_id` and order by the `created_at` to get the info about each customer with its latest order. ```sql SELECT *, json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders GROUP BY customer_id ORDER by created_at LIMIT 5; ``` But there I see the problem, how can we group by customer_id and find the latest order for that customer! The order by will only work after the grouping has been done, so its not necessarily the per order for each customer. Hmm! We need to make a subquery to get the latest order for each customer it seems. ```sql SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag FROM orders WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ORDER BY orders.created_at DESC; ``` We just added this ```sql WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ``` This will filter only the latest created order for each customer and then we can grab the details from json in each row per customer after this isolates the row with the latest created at time. ``` sqlite> SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag FROM orders WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ORDER BY orders.created_at DESC; +-------------+---------------------+-----------------+-----------+-----------+ | customer_id | created_at | shipping_method | gift_wrap | risk_flag | +-------------+---------------------+-----------------+-----------+-----------+ | 32 | 2025-12-17 21:17:39 | overnight | 0 | | | 15 | 2025-12-17 19:21:33 | express | 0 | medium | | 50 | 2025-12-17 14:47:54 | express | 1 | low | | 43 | 2025-12-17 14:23:46 | express | 1 | | | 27 | 2025-12-17 14:05:13 | standard | 1 | | | 3 | 2025-12-17 14:02:28 | standard | 1 | high | | 49 | 2025-12-17 13:28:49 | express | 1 | high | | 36 | 2025-12-17 11:11:29 | overnight | 1 | | | 31 | 2025-12-17 08:05:46 | express | 0 | | | 16 | 2025-12-17 07:32:36 | express | 0 | | | 38 | 2025-12-17 06:05:12 | standard | 1 | | | 44 | 2025-12-17 05:28:54 | standard | 1 | | | 9 | 2025-12-17 04:33:08 | express | 1 | | | 23 | 2025-12-17 03:01:49 | express | 0 | | | 21 | 2025-12-16 23:53:14 | overnight | 1 | | | 25 | 2025-12-16 20:49:58 | overnight | 1 | high | | 46 | 2025-12-16 19:38:37 | standard | 0 | | | 1 | 2025-12-16 19:23:53 | express | 0 | | | 28 | 2025-12-16 18:20:55 | standard | 0 | low | | 40 | 2025-12-16 17:54:05 | express | 0 | | | 13 | 2025-12-16 16:11:16 | standard | 1 | | | 24 | 2025-12-16 14:19:45 | overnight | 0 | | | 11 | 2025-12-16 11:20:31 | standard | 1 | medium | | 17 | 2025-12-16 08:19:36 | standard | 0 | | | 4 | 2025-12-16 04:38:51 | express | 0 | | | 34 | 2025-12-16 02:11:57 | express | 0 | | | 30 | 2025-12-15 15:32:04 | overnight | 0 | medium | | 48 | 2025-12-15 13:03:59 | standard | 1 | | | 41 | 2025-12-15 13:00:00 | standard | 0 | high | | 45 | 2025-12-15 11:37:57 | standard | 0 | | | 7 | 2025-12-14 23:39:47 | express | 0 | | | 35 | 2025-12-14 22:46:36 | express | 1 | high | | 47 | 2025-12-14 20:53:07 | standard | 0 | | | 22 | 2025-12-14 12:38:58 | standard | 0 | medium | | 12 | 2025-12-14 07:59:28 | standard | 1 | medium | | 18 | 2025-12-14 04:55:34 | overnight | 0 | low | | 20 | 2025-12-14 04:54:07 | overnight | 0 | | | 14 | 2025-12-13 07:44:19 | standard | 1 | | | 6 | 2025-12-13 07:03:12 | overnight | 1 | | | 10 | 2025-12-13 04:23:37 | standard | 0 | medium | | 19 | 2025-12-13 03:29:15 | standard | 0 | | | 8 | 2025-12-12 12:42:18 | overnight | 0 | | | 26 | 2025-12-11 17:35:46 | standard | 0 | low | | 37 | 2025-12-11 13:55:35 | overnight | 1 | | | 33 | 2025-12-09 12:30:54 | express | 1 | | | 2 | 2025-12-08 07:17:31 | standard | 0 | | | 42 | 2025-12-08 02:48:12 | overnight | 0 | medium | | 5 | 2025-12-06 17:53:53 | overnight | 1 | | | 39 | 2025-12-06 14:38:29 | overnight | 1 | | | 29 | 2025-12-03 05:10:32 | overnight | 1 | high | +-------------+---------------------+-----------------+-----------+-----------+ sqlite> ``` Now, that is it! I guess, a quick solution for getting this data sorted. However there is one more way without needing the subquery. ### ROW NUMBER - Window Function We can create partition for each customer with the `customer_id` and then sort by the `created_at` date in a latest first order (descending) and simply take the first row for extraction. This is kind of the same thing, but a kind of 'gives a good feel to me' solution. Elite Mindset! ```sql WITH ranked_orders AS ( SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag, ROW_NUMBER() OVER (PARTITION BY orders.customer_id ORDER BY orders.created_at DESC) AS row_num FROM orders ) SELECT customer_id, created_at, shipping_method, gift_wrap, risk_flag FROM ranked_orders WHERE row_num = 1 ORDER BY created_at DESC; ``` We are just doing the same thing as explained. This thing `ROW_NUMBER() OVER (PARTITION BY orders.customer_id ORDER BY orders.created_at DESC) AS row_num` It will partition the table `orders` with `customer_id` and order by `created_at` latest first. And we assign each row as the `row_number` and grab the 1st row per customer to grab the latest order (filter with WHERE when calling the cte, or querying. However we created the CTE `ranked_orders` or you can call it `latest_orders`, to get the `row_num` filter to only the first `1` and for that we grab the already extracted json data as well the other columns. Simple! That is it! Day 9 was easy peasy! Onwards day 10! Catch you tomorrow!

Advent of SQL 2025 Day 8: Product Catalog

Tue, 23 Dec 2025 00:00:00 UTC

## Advent of SQL - Day 8, Product Catalog Whopsies! This is day 8. Let's get straigh... HOOH! We need to clean up some SQL for running in SQLite. ```bash sed -i 's/TIMESTAMP[[:space:]]*//g' day8-inserts-sqlite.sql ``` Just cleaning up `TIMESTAMP` in `INSERT` before the date value. Here we go: The SQL to run in SQLite. ```sql DROP TABLE IF EXISTS price_changes; DROP TABLE IF EXISTS products; CREATE TABLE products ( product_id INT PRIMARY KEY, product_name TEXT ); CREATE TABLE price_changes ( id INT PRIMARY KEY, product_id INT, price NUMERIC(10,2), effective_timestamp ); INSERT INTO products (product_id, product_name) VALUES (1, 'Deluxe Sled'), (2, 'Holiday Trail Mix Trio'), (3, 'Premium Cinnamon Roasted Almonds'), (4, 'Deluxe Wrapping Paper'), (5, 'Deluxe Roasted Cashews'), (6, 'Festive Cookware Set'), (7, 'Deluxe Mug'), (8, 'Premium Sled'), (9, 'Essential Sled'), (10, 'Family Snow Boots'), (11, 'Family Dark Chocolate Almonds'), (12, 'Premium Festive Scarf'), (13, 'Essential Cookie Decorating Kit'), (14, 'Festive White Chocolate Popcorn'), (15, 'Cozy Puzzle'), (16, 'Holiday Cheddar Popcorn'), (17, 'Premium Board Game'), (18, 'Deluxe Pecan Praline Bites'), (19, 'Cozy Almond Brittle'), (20, 'Winter Sled'); INSERT INTO price_changes (id, product_id, price, effective_timestamp) VALUES (1, 1, 148.28, '2025-12-01 05:25:35'), (2, 1, 148.63, '2025-12-02 18:15:33'), (3, 1, 126.78, '2025-12-02 18:40:38'), (4, 1, 119.12, '2025-12-03 10:14:35'), (5, 1, 98.57, '2025-12-04 04:14:31'), (6, 1, 88.49, '2025-12-06 19:02:40'), (7, 1, 80.88, '2025-12-07 10:43:54'), (8, 1, 78.88, '2025-12-08 06:45:39'), (9, 1, 80.24, '2025-12-08 16:11:11'), (10, 1, 73.9, '2025-12-10 14:33:43'), (11, 1, 88.2, '2025-12-12 02:21:09'), (12, 1, 99.03, '2025-12-12 02:58:14'), (13, 1, 100.18, '2025-12-14 15:58:03'), (14, 1, 106.91, '2025-12-16 01:51:05'), (15, 1, 109.25, '2025-12-16 16:01:53'), (16, 2, 29.54, '2025-12-03 14:21:10'), (17, 2, 34.33, '2025-12-03 19:14:31'), (18, 2, 39.08, '2025-12-04 06:13:48'), (19, 2, 32.71, '2025-12-04 18:33:17'), (20, 2, 31.71, '2025-12-05 22:36:14'), (21, 2, 28.88, '2025-12-06 02:42:02'), (22, 2, 23.14, '2025-12-07 09:46:54'), (23, 2, 25.65, '2025-12-07 10:03:45'), (24, 2, 27.06, '2025-12-07 14:39:15'), (25, 2, 24.48, '2025-12-07 20:08:05'), (26, 2, 26.84, '2025-12-09 07:44:32'), (27, 2, 27.39, '2025-12-13 06:25:19'), (28, 2, 26.6, '2025-12-14 10:16:34'), (29, 2, 21.38, '2025-12-15 16:20:20'), (30, 2, 17.75, '2025-12-16 09:28:13'); ``` We can get started. ## Problem > Generate a report, using the products and `price_changes` tables for leadership that returns the `product_name`, `current_price`, `previous_price`, and the difference between the current and previous prices So what we need is - product_name - current_price (latest) - previous_price (just before the latest) - price_difference = current - previous Again we have to meddle with dates, maybe, maybe not! ### With CTEs and JOINs Let's start with the simplest approach. We need 2 prices, the latest(highest date timestamp) and the 2nd latest (the 2nd highest date timestamp). We can get the first pretty easily, but what about the second? Well, if we get the first, then the second should be easy to get right? Right? Because it will be just before it. Well not directly. Let's first grab the max timestamp. ```sql SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id; ``` ``` sqlite> SELECT product_id, MAX(effective_timestamp) AS max_ts FROM price_changes GROUP BY product_id; +------------+---------------------+ | product_id | max_ts | +------------+---------------------+ | 1 | 2025-12-16 16:01:53 | | 2 | 2025-12-16 09:28:13 | | 3 | 2025-12-15 03:20:11 | | 4 | 2025-12-16 01:33:41 | | 5 | 2025-12-12 10:11:48 | | 6 | 2025-12-15 11:31:40 | | 7 | 2025-12-16 03:00:51 | | 8 | 2025-12-15 22:33:48 | | 9 | 2025-12-15 20:05:34 | | 10 | 2025-12-15 20:53:45 | ... ... | 139 | 2025-12-16 04:46:33 | | 140 | 2025-12-16 21:19:30 | | 141 | 2025-12-15 09:50:36 | | 142 | 2025-12-16 18:39:51 | | 143 | 2025-12-15 07:27:06 | | 144 | 2025-12-16 16:25:16 | | 146 | 2025-12-13 07:07:19 | | 148 | 2025-12-16 09:30:11 | | 149 | 2025-12-13 16:40:21 | | 150 | 2025-12-13 08:24:43 | +------------+---------------------+ ``` We got all the timestamp's for each product. But we wanted the prices. Well we can't grab the price here, since we are grouping! We can use the timestamp as we are selecting the `MAX` aggregate function however among the many rows for single product how do we group the price? `MIN`, `MAX`, `AVG`, but that's not we want, we just want the price for that timestamp. Well, we need to join to the same table for that timestamp and grab the price. ```sql SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_ts FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id ) AS latest_price_change ON price_changes.product_id = latest_price_change.product_id AND price_changes.effective_timestamp = latest_price_change.latest_timestamp; ``` Here, we first specify what we want - `product_id` - `current_price` which is the price for the - `latest_timestamp` which is the latest time recorded for the product price. We are grouping this by `product_id` since there are price recorded and various timestamps. We need to get the latest timestamp. So, we do a nested query to join the latest timestamp and join the price with the same timestamp. This condition `price_changes.effective_timestamp = latest_price_change.latest_timestamp` helps us get the `price` for the `latest_timestamp`. We first get each timestamp for the product and then find its max, so that's the inner query from which we joined this table to. Self join with the different thing to find. This gives us 2 things. - Product id - Price for the latest timestamp We don't really want the timestamp in the final result, its just a criteria or a intermediate value to get the current and the previous prices for each product. ``` sqlite> sqlite> SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_ts FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id ) AS latest_price_change ON price_changes.product_id = latest_price_change.product_id AND price_changes.effective_timestamp = latest_price_change.latest_timestamp; +------------+---------------+---------------------+ | product_id | current_price | latest_ts | +------------+---------------+---------------------+ | 1 | 109.25 | 2025-12-16 16:01:53 | | 2 | 17.75 | 2025-12-16 09:28:13 | | 3 | 143.65 | 2025-12-15 03:20:11 | | 4 | 98.51 | 2025-12-16 01:33:41 | | 5 | 124.04 | 2025-12-12 10:11:48 | | 6 | 84.14 | 2025-12-15 11:31:40 | | 7 | 123.09 | 2025-12-16 03:00:51 | | 8 | 221.06 | 2025-12-15 22:33:48 | | 9 | 255.88 | 2025-12-15 20:05:34 | | 10 | 57.99 | 2025-12-15 20:53:45 | ... ... | 139 | 16.41 | 2025-12-16 04:46:33 | | 140 | 173.05 | 2025-12-16 21:19:30 | | 141 | 69.97 | 2025-12-15 09:50:36 | | 142 | 35.05 | 2025-12-16 18:39:51 | | 143 | 153.94 | 2025-12-15 07:27:06 | | 144 | 118.21 | 2025-12-16 16:25:16 | | 146 | 54.73 | 2025-12-13 07:07:19 | | 148 | 107.81 | 2025-12-16 09:30:11 | | 149 | 72.6 | 2025-12-13 16:40:21 | | 150 | 138.66 | 2025-12-13 08:24:43 | +------------+---------------+---------------------+ sqlite> ``` Now, we need to the price before this max timestamp. How do we get it? We need to again join? Yes... We need to subquery the timestamp just before it. But how will we get the max timestamp for each product. Well that's what we wrote above. We can convert that to a CTE. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT * FROM latest; ``` We just got the same thing, however we can now use `latest` as a temporary table in the query. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` So, ok, this is getting long. Just we added this ```sql SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ``` This won't work as we need the latest table which we converted to CTE. So, to get the 2nd highest or latest timestamp for each product, we do this. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id; ``` We use: ```sql JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp ``` To get match the timestamp which will exclude the latest timestamp and that way we can again get the `MAX(price_changes.effective_timestamp) AS previous_timestamp` for the subset of the timestamp. This gives us all the previous timestamps i.e. the price just before the max timestamp for each product. ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id; +------------+----------------+---------------------+ | product_id | previous_price | previous_timestamp | +------------+----------------+---------------------+ | 1 | 106.91 | 2025-12-16 01:51:05 | | 2 | 21.38 | 2025-12-15 16:20:20 | | 3 | 159.65 | 2025-12-14 09:52:09 | | 4 | 105.6 | 2025-12-14 14:09:20 | | 5 | 129.23 | 2025-12-12 04:08:45 | | 6 | 88.97 | 2025-12-15 02:13:04 | | 7 | 127.14 | 2025-12-13 07:25:12 | | 8 | 241.99 | 2025-12-14 19:31:40 | | 9 | 259.56 | 2025-12-12 12:47:13 | | 10 | 64.88 | 2025-12-15 10:52:34 | ... ... | 140 | 157.02 | 2025-12-09 17:58:07 | | 141 | 73.88 | 2025-12-13 14:35:53 | | 142 | 30.25 | 2025-12-16 13:44:42 | | 143 | 143.04 | 2025-12-11 10:08:13 | | 144 | 114.22 | 2025-12-15 17:33:37 | | 146 | 65.71 | 2025-12-10 15:50:14 | | 148 | 101.09 | 2025-12-15 05:37:27 | | 149 | 86.31 | 2025-12-13 13:18:14 | | 150 | 123.61 | 2025-12-12 10:49:22 | +------------+----------------+---------------------+ sqlite> ``` Now, we need to join both of these on the same product id, to fetch both previous and current timestamp as well as the price. ```sql SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` Simple ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; +------------+----------------+---------------------+------------+---------------+---------------------+ | product_id | previous_price | previous_timestamp | product_id | current_price | latest_timestamp | +------------+----------------+---------------------+------------+---------------+---------------------+ | 1 | 106.91 | 2025-12-16 01:51:05 | 1 | 109.25 | 2025-12-16 16:01:53 | | 2 | 21.38 | 2025-12-15 16:20:20 | 2 | 17.75 | 2025-12-16 09:28:13 | | 3 | 159.65 | 2025-12-14 09:52:09 | 3 | 143.65 | 2025-12-15 03:20:11 | | 4 | 105.6 | 2025-12-14 14:09:20 | 4 | 98.51 | 2025-12-16 01:33:41 | | 5 | 129.23 | 2025-12-12 04:08:45 | 5 | 124.04 | 2025-12-12 10:11:48 | | 6 | 88.97 | 2025-12-15 02:13:04 | 6 | 84.14 | 2025-12-15 11:31:40 | | 7 | 127.14 | 2025-12-13 07:25:12 | 7 | 123.09 | 2025-12-16 03:00:51 | | 8 | 241.99 | 2025-12-14 19:31:40 | 8 | 221.06 | 2025-12-15 22:33:48 | | 9 | 259.56 | 2025-12-12 12:47:13 | 9 | 255.88 | 2025-12-15 20:05:34 | | 10 | 64.88 | 2025-12-15 10:52:34 | 10 | 57.99 | 2025-12-15 20:53:45 | ... ... | 139 | 15.26 | 2025-12-12 03:43:32 | 139 | 16.41 | 2025-12-16 04:46:33 | | 140 | 157.02 | 2025-12-09 17:58:07 | 140 | 173.05 | 2025-12-16 21:19:30 | | 141 | 73.88 | 2025-12-13 14:35:53 | 141 | 69.97 | 2025-12-15 09:50:36 | | 142 | 30.25 | 2025-12-16 13:44:42 | 142 | 35.05 | 2025-12-16 18:39:51 | | 143 | 143.04 | 2025-12-11 10:08:13 | 143 | 153.94 | 2025-12-15 07:27:06 | | 144 | 114.22 | 2025-12-15 17:33:37 | 144 | 118.21 | 2025-12-16 16:25:16 | | 146 | 65.71 | 2025-12-10 15:50:14 | 146 | 54.73 | 2025-12-13 07:07:19 | | 148 | 101.09 | 2025-12-15 05:37:27 | 148 | 107.81 | 2025-12-16 09:30:11 | | 149 | 86.31 | 2025-12-13 13:18:14 | 149 | 72.6 | 2025-12-13 16:40:21 | | 150 | 123.61 | 2025-12-12 10:49:22 | 150 | 138.66 | 2025-12-13 08:24:43 | +------------+----------------+---------------------+------------+---------------+---------------------+ sqlite> ``` Now, we are getting somewhere, we just need to find the difference right? Yes, but more JOINs We need the product_name from the `product` table. Almost forgot that table exists? ```sql SELECT * FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` We need both right? So, we need to fetch the product id and join for the latest and the previous table from the CTE, joining on the `product_id`. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM products JOIN latest ON products.product_id = latest.product_id ...> LEFT JOIN previous ON products.product_id = previous.product_id; +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ | product_id | product_name | product_id | current_price | latest_timestamp | product_id | previous_price | previous_timestamp | +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ | 1 | Deluxe Sled | 1 | 109.25 | 2025-12-16 16:01:53 | 1 | 106.91 | 2025-12-16 01:51:05 | | 2 | Holiday Trail Mix Trio | 2 | 17.75 | 2025-12-16 09:28:13 | 2 | 21.38 | 2025-12-15 16:20:20 | | 3 | Premium Cinnamon Roasted Almonds | 3 | 143.65 | 2025-12-15 03:20:11 | 3 | 159.65 | 2025-12-14 09:52:09 | | 4 | Deluxe Wrapping Paper | 4 | 98.51 | 2025-12-16 01:33:41 | 4 | 105.6 | 2025-12-14 14:09:20 | | 5 | Deluxe Roasted Cashews | 5 | 124.04 | 2025-12-12 10:11:48 | 5 | 129.23 | 2025-12-12 04:08:45 | | 6 | Festive Cookware Set | 6 | 84.14 | 2025-12-15 11:31:40 | 6 | 88.97 | 2025-12-15 02:13:04 | | 7 | Deluxe Mug | 7 | 123.09 | 2025-12-16 03:00:51 | 7 | 127.14 | 2025-12-13 07:25:12 | | 8 | Premium Sled | 8 | 221.06 | 2025-12-15 22:33:48 | 8 | 241.99 | 2025-12-14 19:31:40 | | 9 | Essential Sled | 9 | 255.88 | 2025-12-15 20:05:34 | 9 | 259.56 | 2025-12-12 12:47:13 | | 10 | Family Snow Boots | 10 | 57.99 | 2025-12-15 20:53:45 | 10 | 64.88 | 2025-12-15 10:52:34 | ... ... | 140 | Classic Mug | 140 | 173.05 | 2025-12-16 21:19:30 | 140 | 157.02 | 2025-12-09 17:58:07 | | 141 | Family Fruit Assortment | 141 | 69.97 | 2025-12-15 09:50:36 | 141 | 73.88 | 2025-12-13 14:35:53 | | 142 | Classic Ornament | 142 | 35.05 | 2025-12-16 18:39:51 | 142 | 30.25 | 2025-12-16 13:44:42 | | 143 | Essential Ornament | 143 | 153.94 | 2025-12-15 07:27:06 | 143 | 143.04 | 2025-12-11 10:08:13 | | 144 | Premium Trail Mix Trio | 144 | 118.21 | 2025-12-16 16:25:16 | 144 | 114.22 | 2025-12-15 17:33:37 | | 146 | Premium Book Collection | 146 | 54.73 | 2025-12-13 07:07:19 | 146 | 65.71 | 2025-12-10 15:50:14 | | 148 | Cozy Trail Mix Trio | 148 | 107.81 | 2025-12-16 09:30:11 | 148 | 101.09 | 2025-12-15 05:37:27 | | 149 | Family Cheddar Popcorn | 149 | 72.6 | 2025-12-13 16:40:21 | 149 | 86.31 | 2025-12-13 13:18:14 | | 150 | Holiday Headphones | 150 | 138.66 | 2025-12-13 08:24:43 | 150 | 123.61 | 2025-12-12 10:49:22 | +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ sqlite> ``` Now, we just want - product_name - previous_price - current_price - difference of current_price and previous price So, ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT products.product_name, latest.current_price, previous.previous_price, (latest.current_price - previous.previous_price) as price_difference FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT products.product_name, latest.current_price, previous.previous_price, (latest.current_price - previous.previous_price) as price_difference FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; +-----------------------------------+---------------+----------------+-------------------+ | product_name | current_price | previous_price | price_difference | +-----------------------------------+---------------+----------------+-------------------+ | Deluxe Sled | 109.25 | 106.91 | 2.34 | | Holiday Trail Mix Trio | 17.75 | 21.38 | -3.63 | | Premium Cinnamon Roasted Almonds | 143.65 | 159.65 | -16.0 | | Deluxe Wrapping Paper | 98.51 | 105.6 | -7.08999999999999 | | Deluxe Roasted Cashews | 124.04 | 129.23 | -5.18999999999998 | | Festive Cookware Set | 84.14 | 88.97 | -4.83 | | Deluxe Mug | 123.09 | 127.14 | -4.05 | | Premium Sled | 221.06 | 241.99 | -20.93 | | Essential Sled | 255.88 | 259.56 | -3.68000000000001 | ... ... | Classic Ornament | 35.05 | 30.25 | 4.8 | | Essential Ornament | 153.94 | 143.04 | 10.9 | | Premium Trail Mix Trio | 118.21 | 114.22 | 3.98999999999999 | | Premium Book Collection | 54.73 | 65.71 | -10.98 | | Cozy Trail Mix Trio | 107.81 | 101.09 | 6.72 | | Family Cheddar Popcorn | 72.6 | 86.31 | -13.71 | | Holiday Headphones | 138.66 | 123.61 | 15.05 | +-----------------------------------+---------------+----------------+-------------------+ sqlite> ``` Oh ok! That is it! CTEs and some nasty number of JOINs. Can we do it some other ways? Of course we can, and we will ### ROW NUMBER - Window Function We can even use some window functions because we are doing things per product so we can leverage [ROW_NUMBER](https://sqlite.org/windowfunctions.html#:~:text=in%20window%20functions%3A-,row_number(),-The%20number%20of) in our case. > ROW_NUMBER: The number of the row within the current partition. Rows are numbered starting from 1 in the order defined by the ORDER BY clause in the window definition, or in arbitrary order otherwise. So, we can partition by `product_id` in the `price_changes` table and order by the timestamp (latest first i.e. descending) and grab the first two prices. ```sql WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT * FROM ranked_prices; ``` Here, we just partitioned the price table based on the latest to oldest timestamp (descending) and added the other column `row_number` that will be further used to get the `previous` and the `current` price and time. ``` sqlite> WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT * FROM ranked_prices; +------+------------+--------+---------------------+------------+ | id | product_id | price | effective_timestamp | row_number | +------+------------+--------+---------------------+------------+ | 15 | 1 | 109.25 | 2025-12-16 16:01:53 | 1 | | 14 | 1 | 106.91 | 2025-12-16 01:51:05 | 2 | | 13 | 1 | 100.18 | 2025-12-14 15:58:03 | 3 | | 12 | 1 | 99.03 | 2025-12-12 02:58:14 | 4 | | 11 | 1 | 88.2 | 2025-12-12 02:21:09 | 5 | | 10 | 1 | 73.9 | 2025-12-10 14:33:43 | 6 | | 9 | 1 | 80.24 | 2025-12-08 16:11:11 | 7 | | 8 | 1 | 78.88 | 2025-12-08 06:45:39 | 8 | | 7 | 1 | 80.88 | 2025-12-07 10:43:54 | 9 | | 6 | 1 | 88.49 | 2025-12-06 19:02:40 | 10 | | 5 | 1 | 98.57 | 2025-12-04 04:14:31 | 11 | | 4 | 1 | 119.12 | 2025-12-03 10:14:35 | 12 | | 3 | 1 | 126.78 | 2025-12-02 18:40:38 | 13 | | 2 | 1 | 148.63 | 2025-12-02 18:15:33 | 14 | | 1 | 1 | 148.28 | 2025-12-01 05:25:35 | 15 | ... ... | 1260 | 149 | 167.1 | 2025-12-01 06:36:43 | 15 | | 1288 | 150 | 138.66 | 2025-12-13 08:24:43 | 1 | | 1287 | 150 | 123.61 | 2025-12-12 10:49:22 | 2 | | 1286 | 150 | 141.8 | 2025-12-10 07:06:06 | 3 | | 1285 | 150 | 122.16 | 2025-12-09 20:01:54 | 4 | | 1284 | 150 | 122.06 | 2025-12-06 22:27:41 | 5 | | 1283 | 150 | 128.6 | 2025-12-06 13:03:18 | 6 | | 1282 | 150 | 154.72 | 2025-12-05 08:15:08 | 7 | | 1281 | 150 | 170.3 | 2025-12-04 15:25:50 | 8 | | 1280 | 150 | 156.51 | 2025-12-03 19:11:12 | 9 | | 1279 | 150 | 161.93 | 2025-12-02 02:47:10 | 10 | | 1278 | 150 | 174.36 | 2025-12-02 01:39:14 | 11 | | 1277 | 150 | 180.17 | 2025-12-01 20:36:02 | 12 | | 1276 | 150 | 164.35 | 2025-12-01 07:09:29 | 13 | | 1275 | 150 | 141 | 2025-12-01 01:29:46 | 14 | +------+------------+--------+---------------------+------------+ ``` ```sql WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT products.product_name, current_price.price AS current_price, previous_price.price AS previous_price, current_price.price - previous_price.price AS price_difference FROM products LEFT JOIN ranked_prices AS current_price ON products.product_id = current_price.product_id AND current_price.row_number = 1 LEFT JOIN ranked_prices AS previous_price ON products.product_id = previous_price.product_id AND previous_price.row_number = 2; ``` This is pretty simple, we just get - the `current_price` as `row_number = 1` from the `ranked_prices` CTE - the `previous_price` as `row_number = 2` from the `ranked_prices` CTE We got all we needed as: - `products.product_name` - `current_price.price AS current_price` - `previous_price.price AS previous_price` - `current_price.price - previous_price.price AS price_difference` So, this is how `ROW_NUMBER` can be used here. ### With LEAD LAG - Window Functions We can also use [LEAD](https://sqlite.org/windowfunctions.html#:~:text=does%20not%20exist.-,lead(expr),-lead(expr%2C%20offset) and [LAG](https://sqlite.org/windowfunctions.html#:~:text=a%20part%20of.-,lag(expr),-lag(expr%2C%20offset) window functions here. We can specifically use `LAG` as it was a challenge to get the second latest timestamped price. We will partition the `price_changes` table on the `product_id` and order it by the `effective_timestamp`. This will give us the row before the timestamp of the current one, so the first row for each product can be empty (if sorted in ascending order of timestamp since there is no before row for the first row) ```sql WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT * FROM lagged_prices; ``` Here, we simply selected the data that we need, `product_id`, `price` as the `current_price` since we can order by latest timestamp. ``` sqlite> WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT * FROM lagged_prices; +------------+---------------+----------------+---------------------+ | product_id | current_price | previous_price | effective_timestamp | +------------+---------------+----------------+---------------------+ | 1 | 148.28 | | 2025-12-01 05:25:35 | | 1 | 148.63 | 148.28 | 2025-12-02 18:15:33 | | 1 | 126.78 | 148.63 | 2025-12-02 18:40:38 | | 1 | 119.12 | 126.78 | 2025-12-03 10:14:35 | | 1 | 98.57 | 119.12 | 2025-12-04 04:14:31 | | 1 | 88.49 | 98.57 | 2025-12-06 19:02:40 | | 1 | 80.88 | 88.49 | 2025-12-07 10:43:54 | | 1 | 78.88 | 80.88 | 2025-12-08 06:45:39 | | 1 | 80.24 | 78.88 | 2025-12-08 16:11:11 | | 1 | 73.9 | 80.24 | 2025-12-10 14:33:43 | | 1 | 88.2 | 73.9 | 2025-12-12 02:21:09 | | 1 | 99.03 | 88.2 | 2025-12-12 02:58:14 | | 1 | 100.18 | 99.03 | 2025-12-14 15:58:03 | | 1 | 106.91 | 100.18 | 2025-12-16 01:51:05 | | 1 | 109.25 | 106.91 | 2025-12-16 16:01:53 | | 2 | 29.54 | | 2025-12-03 14:21:10 | | 2 | 34.33 | 29.54 | 2025-12-03 19:14:31 | | 2 | 39.08 | 34.33 | 2025-12-04 06:13:48 | | 2 | 32.71 | 39.08 | 2025-12-04 18:33:17 | | 2 | 31.71 | 32.71 | 2025-12-05 22:36:14 | | 2 | 28.88 | 31.71 | 2025-12-06 02:42:02 | ... ... | 149 | 101.03 | 98.4 | 2025-12-10 00:37:46 | | 149 | 95.68 | 101.03 | 2025-12-13 01:10:53 | | 149 | 86.31 | 95.68 | 2025-12-13 13:18:14 | | 149 | 72.6 | 86.31 | 2025-12-13 16:40:21 | | 150 | 141 | | 2025-12-01 01:29:46 | | 150 | 164.35 | 141 | 2025-12-01 07:09:29 | | 150 | 180.17 | 164.35 | 2025-12-01 20:36:02 | | 150 | 174.36 | 180.17 | 2025-12-02 01:39:14 | | 150 | 161.93 | 174.36 | 2025-12-02 02:47:10 | | 150 | 156.51 | 161.93 | 2025-12-03 19:11:12 | | 150 | 170.3 | 156.51 | 2025-12-04 15:25:50 | | 150 | 154.72 | 170.3 | 2025-12-05 08:15:08 | | 150 | 128.6 | 154.72 | 2025-12-06 13:03:18 | | 150 | 122.06 | 128.6 | 2025-12-06 22:27:41 | | 150 | 122.16 | 122.06 | 2025-12-09 20:01:54 | | 150 | 141.8 | 122.16 | 2025-12-10 07:06:06 | | 150 | 123.61 | 141.8 | 2025-12-12 10:49:22 | | 150 | 138.66 | 123.61 | 2025-12-13 08:24:43 | +------------+---------------+----------------+---------------------+ ``` Then we can join. The conditions are important to filter. - we get the lagged time stamp where we want the max of the timestamp since the latest timestamp will have the lagging timestamp for it from the CTE of lagged_prices and current_price as well. ```sql SELECT products.product_name, lagged_prices.current_price, lagged_prices.previous_price, lagged_prices.current_price - lagged_prices.previous_price AS price_difference FROM products JOIN lagged_prices ON products.product_id = lagged_prices.product_id WHERE lagged_prices.effective_timestamp = ( SELECT MAX(effective_timestamp) FROM price_changes WHERE product_id = products.product_id ); ``` ```sql WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT products.product_name, lagged_prices.current_price, lagged_prices.previous_price, lagged_prices.current_price - lagged_prices.previous_price AS price_difference FROM products JOIN lagged_prices ON products.product_id = lagged_prices.product_id WHERE lagged_prices.effective_timestamp = ( SELECT MAX(effective_timestamp) FROM price_changes WHERE product_id = products.product_id ); ``` This yields the same result. This is all big, any smaller queries? Not quite small. ### LIMIT OFFSET and CTE Well here's a little short, but quite dirty. ```sql SELECT *, current_price - previous_price AS price_difference FROM ( SELECT products.product_name, (SELECT price FROM price_changes WHERE price_changes.product_id = products.product_id ORDER BY effective_timestamp DESC LIMIT 1) AS current_price, (SELECT price FROM price_changes WHERE price_changes.product_id = products.product_id ORDER BY effective_timestamp DESC LIMIT 1 OFFSET 1) AS previous_price FROM products ); ``` This basically grabs the price from the latest timestamp and the 2nd timestamp with offset and wraps it in a CTE to compute the difference. Pretty slick if you ask me. But hey! I am done for this! It was a great problem. Getting data behind certain data is quite relatable and challenging enough to explore window functions and what not. That's it from day 8. See you tomorrow for day 9!

Advent of SQL 2025 Day 7: Polar Express Mixin

Mon, 22 Dec 2025 00:00:00 UTC

## Advent of SQL, Day 7 - Polar Express There were a few things, I had to dig up for converting the JSON `ARRAY[]` in the statements into strings for SQLite, we can't really use list of strings in SQLite. Here's the command to convert that array of strings into string. ``` sed "s/ARRAY\['/'\[\"/g; s/','/\",\"/g; s/']/\"]'/g" day7-inserts.sql > day7-inserts-sqlite.sql ``` OK, once that's done, this can be safely run into a sqlite database. ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day7-inserts-sqlite.sql sqlite> .schema CREATE TABLE passengers ( passenger_id INT PRIMARY KEY, passenger_name TEXT, favorite_mixins TEXT[], car_id INT ); CREATE TABLE cocoa_cars ( car_id INT PRIMARY KEY, available_mixins TEXT[], total_stock INT ); sqlite> sqlite> .mode table sqlite> select * from passengers limit 20; +--------------+----------------+--------------------------------------------------------------+--------+ | passenger_id | passenger_name | favorite_mixins | car_id | +--------------+----------------+--------------------------------------------------------------+--------+ | 1 | Ava Johnson | ["vanilla foam"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 5 | Layla Brooks | ["crispy rice","dark chocolate","caramel drizzle","cinnamon" | 3 | | | | ] | | +--------------+----------------+--------------------------------------------------------------+--------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 7 | Sofia Kim | ["cinnamon"] | 9 | +--------------+----------------+--------------------------------------------------------------+--------+ | 8 | Jonah Wolfe | ["cinnamon","dark chocolate"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | +--------------+----------------+--------------------------------------------------------------+--------+ | 10 | Diego Ramos | ["shaved chocolate"] | 1 | +--------------+----------------+--------------------------------------------------------------+--------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 12 | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | 8 | +--------------+----------------+--------------------------------------------------------------+--------+ | 13 | Mila Novak | ["crispy rice","cinnamon"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 14 | Lucas Ford | ["vanilla foam","white chocolate","cinnamon"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 16 | Omar Qureshi | ["marshmallow"] | 3 | +--------------+----------------+--------------------------------------------------------------+--------+ | 17 | Keiko Ito | ["vanilla foam","marshmallow"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 18 | Tariq Hassan | ["dark chocolate","crispy rice","white chocolate","peppermin | 2 | | | | t"] | | +--------------+----------------+--------------------------------------------------------------+--------+ | 19 | Mira Zhao | ["caramel drizzle","marshmallow","cinnamon"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 20 | Bianca Pereira | ["dark chocolate","peppermint"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ sqlite> sqlite> select * from cocoa_cars; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` Here's your full SQL file: ```sql DROP TABLE IF EXISTS passengers; DROP TABLE IF EXISTS cocoa_cars; CREATE TABLE passengers ( passenger_id INT PRIMARY KEY, passenger_name TEXT, favorite_mixins TEXT[], car_id INT ); CREATE TABLE cocoa_cars ( car_id INT PRIMARY KEY, available_mixins TEXT[], total_stock INT ); INSERT INTO passengers (passenger_id, passenger_name, favorite_mixins, car_id) VALUES (1, 'Ava Johnson', '["vanilla foam"]', 2), (2, 'Mateo Cruz', '["caramel drizzle","shaved chocolate","white chocolate"]', 2), (3, 'Nia Grant', '["shaved chocolate"]', 5), (4, 'Hiro Tanaka', '["shaved chocolate"]', 2), (5, 'Layla Brooks', '["crispy rice","dark chocolate","caramel drizzle","cinnamon"]', 3), (6, 'Ravi Patel', '["caramel drizzle","shaved chocolate","white chocolate"]', 5), (7, 'Sofia Kim', '["cinnamon"]', 9), (8, 'Jonah Wolfe', '["cinnamon","dark chocolate"]', 7), (9, 'Elena Morales', '["white chocolate","shaved chocolate","caramel drizzle"]', 6), (10, 'Diego Ramos', '["shaved chocolate"]', 1), (11, 'Zara Sheikh', '["vanilla foam","crispy rice","peppermint"]', 4), (12, 'Caleb Osei', '["shaved chocolate","dark chocolate","white chocolate"]', 8), (13, 'Mila Novak', '["crispy rice","cinnamon"]', 4), (14, 'Lucas Ford', '["vanilla foam","white chocolate","cinnamon"]', 4), (15, 'Yara Haddad', '["white chocolate","dark chocolate"]', 2), (16, 'Omar Qureshi', '["marshmallow"]', 3), (17, 'Keiko Ito', '["vanilla foam","marshmallow"]', 7), (18, 'Tariq Hassan', '["dark chocolate","crispy rice","white chocolate","peppermint"]', 2), (19, 'Mira Zhao', '["caramel drizzle","marshmallow","cinnamon"]', 7), (20, 'Bianca Pereira', '["dark chocolate","peppermint"]', 5), (21, 'Eva Schmidt', '["white chocolate","marshmallow"]', 5), (22, 'Rafael Silva', '["cinnamon","caramel drizzle"]', 3), (23, 'Nolan Murphy', '["caramel drizzle"]', 8), (24, 'Sara Johansson', '["marshmallow"]', 6), (25, 'Ingrid Nilsen', '["shaved chocolate","peppermint","marshmallow"]', 2), (26, 'Arjun Kapoor', '["dark chocolate"]', 2), (27, 'Nova Adams', '["cinnamon","dark chocolate"]', 9), (28, 'Felix Schneider', '["crispy rice","vanilla foam"]', 4), (29, 'Tim Cook', '["crispy rice","vanilla foam"]', 6), (30, 'Sophia Rossi', '["white chocolate","dark chocolate","peppermint","marshmallow"]', 4), (31, 'Liam OConnor', '["caramel drizzle"]', 1), (32, 'Olivia Dubois', '["shaved chocolate","peppermint"]', 2), (33, 'Emma Svensson', '["shaved chocolate"]', 2), (34, 'Noah Fischer', '["caramel drizzle"]', 2), (35, 'William Becker', '["crispy rice","dark chocolate"]', 4), (36, 'Isabella Laurent', '["dark chocolate","shaved chocolate"]', 8), (37, 'James Kim', '["shaved chocolate","marshmallow","peppermint","caramel drizzle"]', 7), (38, 'Mia Chen', '["vanilla foam","white chocolate","shaved chocolate"]', 1), (39, 'Benjamin Patel', '["peppermint"]', 7), (40, 'Charlotte Singh', '["marshmallow","shaved chocolate","white chocolate"]', 4), (41, 'Daniel Murphy', '["cinnamon","vanilla foam","marshmallow","white chocolate"]', 8), (42, 'Zoe Wilson', '["marshmallow","dark chocolate"]', 9), (43, 'Robert Smith', '["peppermint"]', 9), (44, 'Emily Johnson', '["marshmallow"]', 4), (45, 'David Brown', '["vanilla foam","dark chocolate"]', 8), (46, 'Sarah Davis', '["vanilla foam","peppermint"]', 3), (47, 'James Simon', '["peppermint","vanilla foam","cinnamon","shaved chocolate"]', 5), (48, 'Linda Lee', '["white chocolate","dark chocolate","marshmallow","vanilla foam"]', 5), (49, 'Carlos Mendez', '["peppermint","white chocolate"]', 6), (50, 'Fatima Noor', '["peppermint"]', 8), (51, 'Youssef El-Sayed', '["peppermint","marshmallow"]', 3), (52, 'Ian Landsman', '["marshmallow"]', 4), (53, 'Nolan Young', '["marshmallow","shaved chocolate","crispy rice","vanilla foam"]', 1), (54, 'Ava Martinez', '["vanilla foam"]', 9), (55, 'William Chen', '["crispy rice","shaved chocolate","caramel drizzle"]', 6), (56, 'Isabella Rodriguez', '["crispy rice","vanilla foam"]', 3), (57, 'Zachary Collins', '["dark chocolate","caramel drizzle","vanilla foam"]', 2), (58, 'Audrey Edwards', '["dark chocolate"]', 2), (59, 'Jason Stewart', '["white chocolate"]', 4), (60, 'Lucy Morris', '["cinnamon","caramel drizzle","peppermint"]', 4), (61, 'Cameron Rogers', '["crispy rice","cinnamon","shaved chocolate"]', 9), (62, 'Aria Blackwood', '["white chocolate","peppermint","caramel drizzle"]', 9), (63, 'Felix Whitmore', '["marshmallow","cinnamon","dark chocolate"]', 2), (64, 'Luna Hartley', '["white chocolate"]', 3), (65, 'Jasper Thorne', '["crispy rice","shaved chocolate","caramel drizzle"]', 6), (66, 'Nora Calloway', '["crispy rice","dark chocolate"]', 5), (67, 'Silas Merrick', '["marshmallow"]', 7), (68, 'Iris Pembroke', '["peppermint","white chocolate","cinnamon"]', 3), (69, 'Milo Ashford', '["cinnamon","dark chocolate","crispy rice"]', 7), (70, 'Clara Westbrook', '["marshmallow"]', 2), (71, 'Owen Fairchild', '["white chocolate","peppermint"]', 6), (72, 'Ruby Hawthorne', '["vanilla foam","cinnamon"]', 1), (73, 'Finn Lockhart', '["caramel drizzle","peppermint","cinnamon"]', 4), (74, 'Violet Sterling', '["marshmallow","caramel drizzle"]', 9), (75, 'August Blackwell', '["cinnamon","shaved chocolate","vanilla foam","marshmallow"]', 3), (76, 'Hazel Kincaid', '["peppermint","cinnamon","caramel drizzle","dark chocolate"]', 7), (77, 'Leo Greyson', '["crispy rice","cinnamon"]', 2), (78, 'Stella Beaumont', '["peppermint","dark chocolate","marshmallow","vanilla foam"]', 8), (79, 'Miles Brennan', '["crispy rice","shaved chocolate","marshmallow"]', 1), (80, 'Ivy Winslow', '["vanilla foam","caramel drizzle"]', 5), (81, 'Jack Carmichael', '["crispy rice"]', 6), (82, 'Scarlett Dalton', '["white chocolate","caramel drizzle","peppermint","vanilla foam"]', 5), (83, 'Oliver Ashby', '["crispy rice","peppermint"]', 2), (84, 'Aurora Whitfield', '["caramel drizzle","white chocolate","shaved chocolate","crispy rice"]', 9), (85, 'Noah Hastings', '["vanilla foam"]', 4), (86, 'Eliza Radcliffe', '["peppermint","vanilla foam","white chocolate"]', 9), (87, 'Liam Donovan', '["caramel drizzle","vanilla foam"]', 2), (88, 'Penelope Sinclair', '["caramel drizzle","marshmallow","white chocolate"]', 5), (89, 'Ethan Marlowe', '["vanilla foam","caramel drizzle","shaved chocolate"]', 5), (90, 'Charlotte Waverly', '["shaved chocolate","vanilla foam"]', 7), (91, 'Lucas Prescott', '["crispy rice","vanilla foam"]', 9), (92, 'Amelia Rosewood', '["crispy rice"]', 5), (93, 'Henry Treadwell', '["vanilla foam","caramel drizzle"]', 8), (94, 'Sophie Langford', '["dark chocolate","shaved chocolate","crispy rice","white chocolate"]', 3), (95, 'Benjamin Fairweather', '["crispy rice"]', 9), (96, 'Grace Aldridge', '["marshmallow","shaved chocolate","caramel drizzle"]', 5), (97, 'Samuel Kingsley', '["shaved chocolate","cinnamon"]', 1), (98, 'Eleanor Morrison', '["shaved chocolate"]', 8), (99, 'Daniel Lockwood', '["dark chocolate"]', 7), (100, 'Lucy Harrington', '["vanilla foam","dark chocolate"]', 7), (101, 'Matthew Sutherland', '["cinnamon","peppermint"]', 2), (102, 'Emma Gilmore', '["shaved chocolate","cinnamon","vanilla foam","caramel drizzle"]', 9), (103, 'Alexander Stratton', '["peppermint","shaved chocolate","vanilla foam","white chocolate"]', 8), (104, 'Abigail Worthington', '["dark chocolate","caramel drizzle"]', 8), (105, 'William Beauchamp', '["white chocolate","dark chocolate","marshmallow","caramel drizzle"]', 8), (106, 'Hannah Livingston', '["crispy rice","shaved chocolate","vanilla foam","caramel drizzle"]', 5), (107, 'James Garrison', '["shaved chocolate","crispy rice","white chocolate","peppermint"]', 5), (108, 'Sophia Brookshire', '["crispy rice","caramel drizzle"]', 6), (109, 'Theodore Hadley', '["cinnamon"]', 3), (110, 'Olivia Carrington', '["vanilla foam","cinnamon"]', 4), (111, 'Sebastian Ashworth', '["vanilla foam"]', 7), (112, 'Chloe Blackstone', '["white chocolate","dark chocolate","shaved chocolate"]', 1), (113, 'Nicholas Montague', '["vanilla foam","white chocolate"]', 1), (114, 'Madeline Ramsey', '["dark chocolate","peppermint","cinnamon","vanilla foam"]', 7), (115, 'Gabriel Winthrop', '["white chocolate","shaved chocolate","dark chocolate","marshmallow"]', 5), (116, 'Alice Merriweather', '["dark chocolate","peppermint","shaved chocolate","cinnamon"]', 7), (117, 'Isaac Kendrick', '["dark chocolate","cinnamon"]', 9), (118, 'Lillian Holbrook', '["vanilla foam"]', 1), (119, 'Caleb Bellamy', '["vanilla foam"]', 3), (120, 'Rose Drummond', '["cinnamon","peppermint","white chocolate","shaved chocolate"]', 6), (121, 'Elijah Wakefield', '["dark chocolate","caramel drizzle"]', 6), (122, 'Margaret Fairbanks', '["crispy rice","vanilla foam","cinnamon","peppermint"]', 8), (123, 'Julian Blackburn', '["white chocolate"]', 1), (124, 'Eva Templeton', '["shaved chocolate","marshmallow","vanilla foam"]', 1), (125, 'Nathan Whitley', '["shaved chocolate"]', 4), (126, 'Anna Westfield', '["cinnamon"]', 4), (127, 'Aaron Ashcroft', '["dark chocolate","marshmallow"]', 4), (128, 'Julia Pendleton', '["crispy rice","caramel drizzle","marshmallow","white chocolate"]', 2), (129, 'Connor Redmond', '["crispy rice","marshmallow"]', 1), (130, 'Grace Thornhill', '["vanilla foam","white chocolate","caramel drizzle"]', 4), (131, 'Zachary Stafford', '["shaved chocolate"]', 2), (132, 'Caroline Bannister', '["marshmallow","peppermint","cinnamon"]', 9), (133, 'Dylan Blakely', '["caramel drizzle","marshmallow","crispy rice","white chocolate"]', 6), (134, 'Katherine Underwood', '["vanilla foam"]', 8), (135, 'Tyler Braddock', '["vanilla foam"]', 6), (136, 'Victoria Harwood', '["cinnamon","vanilla foam","marshmallow","caramel drizzle"]', 9), (137, 'Ryan Beckett', '["white chocolate","dark chocolate","shaved chocolate"]', 7), (138, 'Elizabeth Chesterfield', '["caramel drizzle","shaved chocolate","vanilla foam"]', 2), (139, 'Jordan Waverly', '["dark chocolate","shaved chocolate","vanilla foam"]', 8), (140, 'Sarah Remington', '["caramel drizzle","peppermint","shaved chocolate","cinnamon"]', 3), (141, 'Brandon Locklear', '["shaved chocolate","caramel drizzle","vanilla foam","cinnamon"]', 6), (142, 'Rachel Wyndham', '["crispy rice","peppermint","white chocolate"]', 4), (143, 'Logan Sherwood', '["shaved chocolate"]', 7), (144, 'Amanda Fitzroy', '["white chocolate","shaved chocolate","caramel drizzle","dark chocolate"]', 8), (145, 'Jackson Thorpe', '["peppermint","marshmallow","cinnamon","dark chocolate"]', 7), (146, 'Rebecca Ashcombe', '["crispy rice","caramel drizzle"]', 8), (147, 'Cameron Gladstone', '["vanilla foam","caramel drizzle","cinnamon"]', 8), (148, 'Jessica Langston', '["crispy rice","white chocolate","marshmallow"]', 2), (149, 'Mason Fairmont', '["caramel drizzle","marshmallow"]', 9), (150, 'Emily Claridge', '["shaved chocolate","white chocolate"]', 8), (151, 'Hunter Bellingham', '["white chocolate","crispy rice","peppermint"]', 4), (152, 'Laura Thornbury', '["shaved chocolate","caramel drizzle","marshmallow"]', 5), (153, 'Wyatt Alderton', '["white chocolate"]', 3), (154, 'Claire Berkshire', '["peppermint","white chocolate","crispy rice"]', 5), (155, 'Cole Ashland', '["dark chocolate","marshmallow"]', 1), (156, 'Diana Brightwell', '["dark chocolate","white chocolate","shaved chocolate"]', 6), (157, 'Aiden Stanfield', '["peppermint","crispy rice"]', 8), (158, 'Natalie Warwick', '["marshmallow"]', 7), (159, 'Parker Blackmore', '["marshmallow","peppermint","white chocolate","vanilla foam"]', 5), (160, 'Morgan Steadman', '["shaved chocolate"]', 2), (161, 'Blake Dunwood', '["shaved chocolate","vanilla foam","white chocolate","dark chocolate"]', 5), (162, 'Taylor Woodridge', '["crispy rice","peppermint","white chocolate","caramel drizzle"]', 2), (163, 'Chase Ashbury', '["shaved chocolate","crispy rice"]', 2), (164, 'Madison Clearwater', '["shaved chocolate","cinnamon"]', 9), (165, 'Carter Brookfield', '["cinnamon"]', 1), (166, 'Ashley Fairhaven', '["dark chocolate","white chocolate","cinnamon","peppermint"]', 4), (167, 'Griffin Hartwell', '["crispy rice","dark chocolate","peppermint"]', 4), (168, 'Megan Redfield', '["shaved chocolate","vanilla foam","peppermint"]', 9), (169, 'Grayson Westmore', '["cinnamon","crispy rice"]', 3), (170, 'Nicole Ashridge', '["peppermint"]', 3), (171, 'Sawyer Hollingsworth', '["crispy rice","dark chocolate","peppermint"]', 8), (172, 'Alexis Thorndale', '["vanilla foam","crispy rice","dark chocolate"]', 9), (173, 'Declan Summerfield', '["dark chocolate","marshmallow","crispy rice","peppermint"]', 7), (174, 'Samantha Brightwood', '["crispy rice","peppermint","dark chocolate"]', 4), (175, 'Tristan Ashbrook', '["crispy rice"]', 1), (176, 'Melissa Ravenscroft', '["dark chocolate","white chocolate"]', 5), (177, 'Colton Hawthorne', '["vanilla foam","dark chocolate"]', 2), (178, 'Lauren Silverton', '["caramel drizzle","vanilla foam","cinnamon","dark chocolate"]', 2), (179, 'Landon Whitworth', '["caramel drizzle","vanilla foam"]', 8), (180, 'Kayla Mansfield', '["vanilla foam","peppermint","shaved chocolate"]', 2); INSERT INTO cocoa_cars (car_id, available_mixins, total_stock) VALUES (5, '["white chocolate","shaved chocolate"]', 412), (2, '["cinnamon","marshmallow","caramel drizzle"]', 359), (9, '["crispy rice","peppermint","caramel drizzle","shaved chocolate"]', 354), (4, '["shaved chocolate","white chocolate"]', 338), (8, '["vanilla foam","marshmallow"]', 263), (1, '["peppermint","crispy rice"]', 205), (6, '["shaved chocolate","dark chocolate","crispy rice","cinnamon","peppermint"]', 161), (7, '["caramel drizzle","crispy rice","marshmallow","vanilla foam","cinnamon"]', 132), (3, '["vanilla foam","peppermint"]', 95); ``` Changing `TEXT[]` to `TEXT` is not required, as SQLITE really doesn't bother with the types unless the table is strict. We'll take that flexibility here. ## Problem > Get the stewards a list of all the passengers and the cocoa car(s) they can be served from that has at least one of their favorite mixins. > > Remember only the top three most-stocked cocoa cars remained operational, so the passengers must be served from one of those cars. Ok so, we have two tables. 1. `cocoa_cars` 2. `passengers` What we need to do is to list down the passengers which the `cocoa_cars` can satisfy their atleast one mixin(food) with their car_ids. So, there are limited cars, just 9 cars in total right? ```sql SELECT * FROM cocoa_cars ORDER BY car_id; ``` So, there are only 9 cars. ``` sqlite> SELECT * FROM cocoa_cars ORDER BY car_id; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` However, the question states, we only need to consider the top three most stacked cars. Now we can order by `total_stack` here and find the top stacked 3 cars with the `LIMIT 3` clause. ```sql SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3; ``` ``` sqlite> sqlite> select * from cocoa_cars ORDER BY total_stock DESC; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ sqlite> select * from cocoa_cars ORDER BY total_stock DESC LIMIT 3; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` Now, we have all the cars that we can start assigning the passengers. How? We need to select and assign the passenger the `car_id` which contains one or more of their `favourite_mixins`. Now, this is the tricky part. We are in SQLite! We have a `favourite_list` for `Mateo Cruz` as a string like `["caramel drizzle","shaved chocolate","white chocolate"]` which we need to match against these 3 car `available_mixins`: - `["white chocolate","shaved chocolate"]` on car_id `5` - `["cinnamon","marshmallow","caramel drizzle"]` on car_id `2` - `["crispy rice","peppermint","caramel drizzle","shaved chocolate"]` on car_id `9` So here all three cars have at least one right? - car_id `5` has `shaved_chocolate` - car_id `2` has `caramel drizzle` - car_id `9` has 2 of his 3 `favourite_list` dishes `caramel drizzle`, and `shaved chocolate"`. So we should ideally return for `Matro Cruz` the car_ids `[5, 2, 9]` or as separate rows doesn't matter as much I think. But the first one looks cool! So, how do we do it? First let's start with what we had! The top 3 cars as ```sql SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3; ``` This gives us the table that we can use to compare for each car_id if we have any one of the `favourite_list` for one passenger at a time. But how do we split the string of mixins and favourite_list? We can use the [json_each](https://sqlite.org/json1.html#jeach) function which takes in any valid json string (could be raw, could be a column name) for each row of the the table. And it returns back a lot of things. Let's just try to select everything from the `json_each` with the favourite_mixins column. ```sql SELECT * FROM json_each(favorite_mixins) FROM passengers; ``` ```sqlite> SELECT * FROM json_each(favorite_mixins) FROM passengers; Parse error: near "FROM": syntax error SELECT * FROM json_each(favorite_mixins) FROM passengers; error here ---^ sqlite> ``` Ops! Why? Because we have given it the whole column, it can only take one cell at a time, so we need to give it that cell for each row. ```sql SELECT * FROM json_each((SELECT favorite_mixins FROM passengers)); ``` Here we try to give it only the `favorite_mixins` from the passengers column. ``` sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers)); +-----+--------------+------+--------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+--------------+------+--------------+----+--------+---------+------+ | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +-----+--------------+------+--------------+----+--------+---------+------+ sqlite> select * from passengers limit 5; +--------------+----------------+--------------------------------------------------------------+--------+ | passenger_id | passenger_name | favorite_mixins | car_id | +--------------+----------------+--------------------------------------------------------------+--------+ | 1 | Ava Johnson | ["vanilla foam"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 5 | Layla Brooks | ["crispy rice","dark chocolate","caramel drizzle","cinnamon" | 3 | | | | ] | | +--------------+----------------+--------------------------------------------------------------+--------+ sqlite> ``` But it only gave it for the first column, unluckily it had just one mixin. What is happening? Well! This actually selects only the first row's `favorite_mixins` because `json_each()` processes one value at a time. Since favorite_mixins is a JSON array, SQLite expects a single array value per row. When we try to pass the entire column, it only processes the first row of `favorite_mixins` Let's try to use the json_each for passengers with more than one favorite mixins. ```sql SELECT * FROM json_each( (SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Mateo Cruz') ); ``` ``` sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Ava Johnson')); +-----+--------------+------+--------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+--------------+------+--------------+----+--------+---------+------+ | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +-----+--------------+------+--------------+----+--------+---------+------+ sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Mateo Cruz')); +-----+------------------+------+------------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+------------------+------+------------------+----+--------+---------+------+ | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | +-----+------------------+------+------------------+----+--------+---------+------+ sqlite> ``` Ok now that is neat, it returned 3 rows for the 3 favorite items for the passenger `Mateo Cruz` Now what? How do we get it for all the passengers? How about JOINs, since the passenger's data will remain the same, we just change the mixins for each of their favorite list. ```sql SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; ``` ``` sqlite> sqlite> SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | passenger_id | passenger_name | favorite_mixins | car_id | key | value | type | atom | id | parent | fullkey | path | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 127 | Aaron Ashcroft | ["dark chocolate","marshmallow"] | 4 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 127 | Aaron Ashcroft | ["dark chocolate","marshmallow"] | 4 | 1 | marshmallow | text | marshmallow | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 104 | Abigail Worthington | ["dark chocolate","caramel drizzle"] | 8 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 104 | Abigail Worthington | ["dark chocolate","caramel drizzle"] | 8 | 1 | caramel drizzle | text | caramel drizzle | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 157 | Aiden Stanfield | ["peppermint","crispy rice"] | 8 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 157 | Aiden Stanfield | ["peppermint","crispy rice"] | 8 | 1 | crispy rice | text | crispy rice | 13 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | | | | late"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 1 | shaved chocolate | text | shaved chocolate | 13 | | $[1] | $ | | | | late"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 2 | vanilla foam | text | vanilla foam | 31 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ ... ... | 153 | Wyatt Alderton | ["white chocolate"] | 3 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | 1 | dark chocolate | text | dark chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 51 | Youssef El-Sayed | ["peppermint","marshmallow"] | 3 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 51 | Youssef El-Sayed | ["peppermint","marshmallow"] | 3 | 1 | marshmallow | text | marshmallow | 13 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 1 | caramel drizzle | text | caramel drizzle | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 2 | vanilla foam | text | vanilla foam | 35 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 131 | Zachary Stafford | ["shaved chocolate"] | 2 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 1 | crispy rice | text | crispy rice | 16 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 2 | peppermint | text | peppermint | 28 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 42 | Zoe Wilson | ["marshmallow","dark chocolate"] | 9 | 0 | marshmallow | text | marshmallow | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 42 | Zoe Wilson | ["marshmallow","dark chocolate"] | 9 | 1 | dark chocolate | text | dark chocolate | 14 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ ``` It just automagically joins the relevant rows for the expanded rows from the json_each when we pass the column which is a json list. But we don't want everything, we just want the passenger and the mixin names. And let's also include the index just to see the data. ```sql SELECT passengers.passenger_name, mixin.key AS mixin_index, mixin.value AS mixin FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; ``` ``` sqlite> SELECT passengers.passenger_name, mixin.key AS mixin_index, mixin.value AS mixin FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; +------------------------+-------------+------------------+ | passenger_name | mixin_index | mixin | +------------------------+-------------+------------------+ | Aaron Ashcroft | 0 | dark chocolate | | Aaron Ashcroft | 1 | marshmallow | | Abigail Worthington | 0 | dark chocolate | | Abigail Worthington | 1 | caramel drizzle | | Aiden Stanfield | 0 | peppermint | | Aiden Stanfield | 1 | crispy rice | | Alexander Stratton | 0 | peppermint | | Alexander Stratton | 1 | shaved chocolate | | Alexander Stratton | 2 | vanilla foam | | Alexander Stratton | 3 | white chocolate | ... ... | Yara Haddad | 1 | dark chocolate | | Youssef El-Sayed | 0 | peppermint | | Youssef El-Sayed | 1 | marshmallow | | Zachary Collins | 0 | dark chocolate | | Zachary Collins | 1 | caramel drizzle | | Zachary Collins | 2 | vanilla foam | | Zachary Stafford | 0 | shaved chocolate | | Zara Sheikh | 0 | vanilla foam | | Zara Sheikh | 1 | crispy rice | | Zara Sheikh | 2 | peppermint | | Zoe Wilson | 0 | marshmallow | | Zoe Wilson | 1 | dark chocolate | +------------------------+-------------+------------------+ ``` That looks good. Now what? We also need to do it for the cocoa_cars with `available_mixins` ```sql SELECT * FROM cocoa_cars JOIN json_each(available_mixins) AS mixin ORDER BY total_stock DESC; ``` OR ```sql SELECT car_id, mixin.key AS mixin_index, mixin.value AS mixin FROM cocoa_cars JOIN json_each(available_mixins) as mixin ORDER BY total_stock DESC; ``` ``` sqlite> SELECT car_id, mixin.key as mixin_index, mixin.value as mixin from cocoa_cars join json_each(available_mixins) as mixin ORDER BY total_stock DESC; +--------+-------------+------------------+ | car_id | mixin_index | mixin | +--------+-------------+------------------+ | 5 | 0 | white chocolate | | 5 | 1 | shaved chocolate | | 2 | 0 | cinnamon | | 2 | 1 | marshmallow | | 2 | 2 | caramel drizzle | | 9 | 0 | crispy rice | | 9 | 1 | peppermint | | 9 | 2 | caramel drizzle | | 9 | 3 | shaved chocolate | | 4 | 0 | shaved chocolate | | 4 | 1 | white chocolate | | 8 | 0 | vanilla foam | | 8 | 1 | marshmallow | | 1 | 0 | peppermint | | 1 | 1 | crispy rice | | 6 | 0 | shaved chocolate | | 6 | 1 | dark chocolate | | 6 | 2 | crispy rice | | 6 | 3 | cinnamon | | 6 | 4 | peppermint | | 7 | 0 | caramel drizzle | | 7 | 1 | crispy rice | | 7 | 2 | marshmallow | | 7 | 3 | vanilla foam | | 7 | 4 | cinnamon | | 3 | 0 | vanilla foam | | 3 | 1 | peppermint | +--------+-------------+------------------+ sqlite> ``` But hold on! We only needed it for the top 3 stocked cars. So, how do we do it, we can (bad and dirty practise) limit by the count of the numbers of mixin in 5, 2 and 9 but that is bad. We need to dynamically get this table. Help in! CTEs ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM stocked_cars JOIN json_each(stocked_cars.available_mixins); ``` OR ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT car_id, car_mixins.key as car_mixin_index, car_mixins.value as car_mixin FROM stocked_cars JOIN json_each(stocked_cars.available_mixins) as car_mixins; ``` ``` sqlite> WITH stocked_cars as ( (x1...> select * from cocoa_cars ORDER BY total_stock DESC LIMIT 3) ...> select * from stocked_cars JOIN json_each(stocked_cars.available_mixins); +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | car_id | available_mixins | total_stock | key | value | type | atom | id | parent | fullkey | path | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 0 | cinnamon | text | cinnamon | 2 | | $[0] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 1 | marshmallow | text | marshmallow | 11 | | $[1] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 2 | caramel drizzle | text | caramel drizzle | 23 | | $[2] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 3 | shaved chocolate | text | shaved chocolate | 42 | | $[3] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT car_id, car_mixins.key as car_mixin_index, car_mixins.value as car_mixin FROM stocked_cars JOIN json_each(stocked_cars.available_mixins) as car_mixins; +--------+-----------------+------------------+ | car_id | car_mixin_index | car_mixin | +--------+-----------------+------------------+ | 5 | 0 | white chocolate | | 5 | 1 | shaved chocolate | | 2 | 0 | cinnamon | | 2 | 1 | marshmallow | | 2 | 2 | caramel drizzle | | 9 | 0 | crispy rice | | 9 | 1 | peppermint | | 9 | 2 | caramel drizzle | | 9 | 3 | shaved chocolate | +--------+-----------------+------------------+ sqlite> ``` Now what next? We simply need to combine all of it - Grab the top 3 cars - Grab the passenger favorite mixins (expand with json_each) - Grab the car available mixins (expand with json_each and use the top 3 car as cte) - JOIN them when the car_mixin has one mixin from a passenger favorite mixin. ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` OR Select some more rows for visual confirmation. ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, passengers.favorite_mixins, stocked_cars.available_mixins, passenger_mixin.value as passenger_mixin, available_mixin.value as available_mixin, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` OR select everything ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` ``` sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +------------------------+--------+ | passenger_name | car_id | +------------------------+--------+ | Mateo Cruz | 5 | | Mateo Cruz | 5 | | Nia Grant | 5 | | Hiro Tanaka | 5 | | Ravi Patel | 5 | | Ravi Patel | 5 | | Elena Morales | 5 | | Elena Morales | 5 | | Diego Ramos | 5 | | Caleb Osei | 5 | | Caleb Osei | 5 | | Lucas Ford | 5 | | Yara Haddad | 5 | | Tariq Hassan | 5 | | Eva Schmidt | 5 | | Ingrid Nilsen | 5 | | Sophia Rossi | 5 | | Olivia Dubois | 5 | | Emma Svensson | 5 | | Isabella Laurent | 5 | | James Kim | 5 | ... ... | Griffin Hartwell | 9 | | Griffin Hartwell | 9 | | Megan Redfield | 9 | | Megan Redfield | 9 | | Grayson Westmore | 9 | | Nicole Ashridge | 9 | | Sawyer Hollingsworth | 9 | | Sawyer Hollingsworth | 9 | | Alexis Thorndale | 9 | | Declan Summerfield | 9 | | Declan Summerfield | 9 | | Samantha Brightwood | 9 | | Samantha Brightwood | 9 | | Tristan Ashbrook | 9 | | Lauren Silverton | 9 | | Landon Whitworth | 9 | | Kayla Mansfield | 9 | | Kayla Mansfield | 9 | +------------------------+--------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, passengers.favorite_mixins, stocked_cars.available_mixins, passenger_mixin.value as passenger_mixin, available_mixin.value as available_mixin, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | passenger_name | favorite_mixins | available_mixins | passenger_mixin | available_mixin | car_id | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Nia Grant | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Hiro Tanaka | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Diego Ramos | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Lucas Ford | ["vanilla foam","white chocolate","cinnamon"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Yara Haddad | ["white chocolate","dark chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Tariq Hassan | ["dark chocolate","crispy rice","white chocolate","peppermin | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | | | t"] | | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Eva Schmidt | ["white chocolate","marshmallow"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ ... ... +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Alexis Thorndale | ["vanilla foam","crispy rice","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Tristan Ashbrook | ["crispy rice"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Lauren Silverton | ["caramel drizzle","vanilla foam","cinnamon","dark chocolate | ["crispy rice","peppermint","caramel drizzle","shaved chocol | caramel drizzle | caramel drizzle | 9 | | | "] | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Landon Whitworth | ["caramel drizzle","vanilla foam"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | caramel drizzle | caramel drizzle | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | shaved chocolate | shaved chocolate | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | passenger_id | passenger_name | favorite_mixins | car_id | key | value | type | atom | id | parent | fullkey | path | car_id | available_mixins | total_stock | key | value | type | atom | id | parent | fullkey | path | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 10 | Diego Ramos | ["shaved chocolate"] | 1 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | ... ... +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 171 | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | 8 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 171 | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | 8 | 2 | peppermint | text | peppermint | 30 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 172 | Alexis Thorndale | ["vanilla foam","crispy rice","dark chocolate"] | 9 | 1 | crispy rice | text | crispy rice | 16 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 173 | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | 7 | 2 | crispy rice | text | crispy rice | 30 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 173 | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | 7 | 3 | peppermint | text | peppermint | 42 | | $[3] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 174 | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | 4 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 174 | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | 4 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 175 | Tristan Ashbrook | ["crispy rice"] | 1 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 178 | Lauren Silverton | ["caramel drizzle","vanilla foam","cinnamon","dark chocolate | 2 | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | | "] | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 179 | Landon Whitworth | ["caramel drizzle","vanilla foam"] | 8 | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 180 | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | 2 | 1 | peppermint | text | peppermint | 16 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 180 | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | 2 | 2 | shaved chocolate | text | shaved chocolate | 27 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 3 | shaved chocolate | text | shaved chocolate | 42 | | $[3] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ sqlite> ``` Now what we did here? - Create the top 3 stocked cars as CTE - The main query first fetches each passenger with its favorite mixin expanded - It joined the table of available_mixin when? - The mixin from passenger is equal to the mixin in the available mixin in the car - Hence we get the car_id as the mixin served for that passenger. You can see we have multiple rows for each user, we might not want that. Though nothing is wrong with it, but the report looks quite long, especially if there were more than a couple of mixins for the passenger or the number of cars were more. We can group by the `passenger_name` and sort of concatenate the ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' AS cocoa_cars FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value GROUP BY passengers.passenger_name; ``` Here, we have added, ``` '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' ``` AND to group by ``` GROUP BY passengers.passenger_name; ``` So this will squish down all the separate rows of the passenger favorite mixins and the cars that have it, with a single row and we defined how we want to squish the different `car_id`s with a [GROUP_CONCAT](https://www.sqlite.org/lang_aggfunc.html#group_concat). This function can concatentate (join or attach or combine) together the multiple strings with a specific separator (by default the separate is `,`). We also use the `||` concatenation operator to add `[` at the start and `]` at the end of the list of `car_ids`. > The group_concat() function returns a string which is the concatenation of all non-NULL values of X. If parameter Y is present then it is used as the separator between instances of X. A comma (",") is used as the separator if Y is omitted. ``` sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' AS cocoa_cars FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value ...> GROUP BY passengers.passenger_name; +------------------------+------------+ | passenger_name | cocoa_cars | +------------------------+------------+ | Aaron Ashcroft | [2] | | Abigail Worthington | [2,9] | | Aiden Stanfield | [9] | | Alexander Stratton | [5,9] | | Alexis Thorndale | [9] | | Alice Merriweather | [5,2,9] | | Amanda Fitzroy | [5,2,9] | | Amelia Rosewood | [9] | | Anna Westfield | [2] | | Aria Blackwood | [5,2,9] | | Ashley Fairhaven | [5,2,9] | | August Blackwell | [5,2,9] | | Aurora Whitfield | [5,2,9] | | Benjamin Fairweather | [9] | | Benjamin Patel | [9] | | Bianca Pereira | [9] | | Blake Dunwood | [5,9] | | Brandon Locklear | [5,2,9] | | Caleb Osei | [5,9] | | Cameron Gladstone | [2,9] | | Cameron Rogers | [5,2,9] | | Carlos Mendez | [5,9] | | Caroline Bannister | [2,9] | | Carter Brookfield | [2] | | Charlotte Singh | [5,2,9] | | Charlotte Waverly | [5,9] | | Chase Ashbury | [5,9] | | Chloe Blackstone | [5,9] | | Claire Berkshire | [5,9] | | Clara Westbrook | [2] | | Cole Ashland | [2] | | Connor Redmond | [2,9] | | Daniel Murphy | [5,2] | | Declan Summerfield | [2,9] | | Diana Brightwell | [5,9] | | Diego Ramos | [5,9] | | Dylan Blakely | [5,2,9] | | Eleanor Morrison | [5,9] | | Elena Morales | [5,2,9] | | Elijah Wakefield | [2,9] | | Eliza Radcliffe | [5,9] | | Elizabeth Chesterfield | [5,2,9] | | Emily Claridge | [5,9] | | Emily Johnson | [2] | | Emma Gilmore | [5,2,9] | | Emma Svensson | [5,9] | | Ethan Marlowe | [5,2,9] | | Eva Schmidt | [5,2] | | Eva Templeton | [5,2,9] | | Fatima Noor | [9] | | Felix Schneider | [9] | | Felix Whitmore | [2] | | Finn Lockhart | [2,9] | | Gabriel Winthrop | [5,2,9] | | Grace Aldridge | [5,2,9] | | Grace Thornhill | [5,2,9] | | Grayson Westmore | [2,9] | | Griffin Hartwell | [9] | | Hannah Livingston | [5,2,9] | | Hazel Kincaid | [2,9] | | Henry Treadwell | [2,9] | | Hiro Tanaka | [5,9] | | Hunter Bellingham | [5,9] | | Ian Landsman | [2] | | Ingrid Nilsen | [5,2,9] | | Iris Pembroke | [5,2,9] | | Isaac Kendrick | [2] | | Isabella Laurent | [5,9] | | Isabella Rodriguez | [9] | | Ivy Winslow | [2,9] | | Jack Carmichael | [9] | | Jackson Thorpe | [2,9] | | James Garrison | [5,9] | | James Kim | [5,2,9] | | James Simon | [5,2,9] | | Jason Stewart | [5] | | Jasper Thorne | [5,2,9] | | Jessica Langston | [5,2,9] | | Jonah Wolfe | [2] | | Jordan Waverly | [5,9] | | Julia Pendleton | [5,2,9] | | Julian Blackburn | [5] | | Kayla Mansfield | [5,9] | | Keiko Ito | [2] | | Landon Whitworth | [2,9] | | Laura Thornbury | [5,2,9] | | Lauren Silverton | [2,9] | | Layla Brooks | [2,9] | | Leo Greyson | [2,9] | | Liam Donovan | [2,9] | | Liam OConnor | [2,9] | | Linda Lee | [5,2] | | Logan Sherwood | [5,9] | | Lucas Ford | [5,2] | | Lucas Prescott | [9] | | Lucy Morris | [2,9] | | Luna Hartley | [5] | | Madeline Ramsey | [2,9] | | Madison Clearwater | [5,2,9] | | Margaret Fairbanks | [2,9] | | Mason Fairmont | [2,9] | | Mateo Cruz | [5,2,9] | | Matthew Sutherland | [2,9] | | Megan Redfield | [5,9] | | Melissa Ravenscroft | [5] | | Mia Chen | [5,9] | | Mila Novak | [2,9] | | Miles Brennan | [5,2,9] | | Milo Ashford | [2,9] | | Mira Zhao | [2,9] | | Morgan Steadman | [5,9] | | Natalie Warwick | [2] | | Nathan Whitley | [5,9] | | Nia Grant | [5,9] | | Nicholas Montague | [5] | | Nicole Ashridge | [9] | | Noah Fischer | [2,9] | | Nolan Murphy | [2,9] | | Nolan Young | [5,2,9] | | Nora Calloway | [9] | | Nova Adams | [2] | | Oliver Ashby | [9] | | Olivia Carrington | [2] | | Olivia Dubois | [5,9] | | Omar Qureshi | [2] | | Owen Fairchild | [5,9] | | Parker Blackmore | [5,2,9] | | Penelope Sinclair | [5,2,9] | | Rachel Wyndham | [5,9] | | Rafael Silva | [2,9] | | Ravi Patel | [5,2,9] | | Rebecca Ashcombe | [2,9] | | Robert Smith | [9] | | Rose Drummond | [5,2,9] | | Ruby Hawthorne | [2] | | Ryan Beckett | [5,9] | | Samantha Brightwood | [9] | | Samuel Kingsley | [5,2,9] | | Sara Johansson | [2] | | Sarah Davis | [9] | | Sarah Remington | [5,2,9] | | Sawyer Hollingsworth | [9] | | Scarlett Dalton | [5,2,9] | | Silas Merrick | [2] | | Sofia Kim | [2] | | Sophia Brookshire | [2,9] | | Sophia Rossi | [5,2,9] | | Sophie Langford | [5,9] | | Stella Beaumont | [2,9] | | Tariq Hassan | [5,9] | | Taylor Woodridge | [5,2,9] | | Theodore Hadley | [2] | | Tim Cook | [9] | | Tristan Ashbrook | [9] | | Victoria Harwood | [2,9] | | Violet Sterling | [2,9] | | William Beauchamp | [5,2,9] | | William Becker | [9] | | William Chen | [5,2,9] | | Wyatt Alderton | [5] | | Yara Haddad | [5] | | Youssef El-Sayed | [2,9] | | Zachary Collins | [2,9] | | Zachary Stafford | [5,9] | | Zara Sheikh | [9] | | Zoe Wilson | [2] | +------------------------+------------+ sqlite> ``` I think we are done. We did it, it was a bit different one. Some wired hack here and there but we made it! Day 7 done and dusted. On to the day 8.

Advent of SQL 2025 Day 6: Days of Delight

Sun, 21 Dec 2025 00:00:00 UTC

## Advent of SQL Day 6: Days of Delight It is day 6 of advent of SQL. Let's jump straight into the sql for the day. ```sql DROP TABLE IF EXISTS families; DROP TABLE IF EXISTS deliveries_assigned; CREATE TABLE families ( id INT PRIMARY KEY, family_name TEXT ); CREATE TABLE deliveries_assigned ( id INT PRIMARY KEY, family_id INT, gift_date DATE, gift_name TEXT ); INSERT INTO families (id, family_name) VALUES (1, 'Isla Martinez'), (2, 'Nolan Garcia'), (3, 'Yara Chen'), (4, 'Tariq Nguyen'), (5, 'Mila Hernandez'), (6, 'Casey Kim'), (7, 'Mateo Hernandez'), (8, 'Keiko Petrov'), (9, 'Ethan Flores'), (10, 'Mateo Nakamura'), (11, 'Maya Fernandez'), (12, 'Mila Davis'), (13, 'Yara Rossi'), (14, 'Nolan Phillips'), (15, 'Amina Perez'); INSERT INTO deliveries_assigned (id, family_id, gift_date, gift_name) VALUES (1, 1, '2025-12-01', 'roasted cashews'), (2, 1, '2025-12-02', 'cookie decorating kit'), (3, 1, '2025-12-03', 'dark chocolate assortment'), (4, 1, '2025-12-04', 'white chocolate candies'), (5, 1, '2025-12-05', 'reindeer headband'), (6, 1, '2025-12-06', 'holiday brownie bites'), (7, 1, '2025-12-07', 'shortbread cookie tin'), (8, 1, '2025-12-08', 'chocolate chip cookies'), (9, 1, '2025-12-11', 'holiday jam trio'), (10, 1, '2025-12-12', 'white chocolate popcorn'), (11, 1, '2025-12-14', 'holiday jam trio'), (12, 1, '2025-12-15', 'fudge bites'), (13, 1, '2025-12-16', 'holiday sticker sheet'), (14, 1, '2025-12-18', 'hot cocoa bombs'), (15, 1, '2025-12-19', 'honey roasted nuts'), (16, 1, '2025-12-20', 'holiday mug'), (17, 1, '2025-12-21', 'white chocolate candies'), (18, 1, '2025-12-22', 'puzzle book'), (19, 1, '2025-12-23', 'snowman plush'), (20, 1, '2025-12-24', 'scented hand cream'), (21, 1, '2025-12-25', 'vanilla bean wafers'), (22, 2, '2025-12-01', 'roasted cashews'), (23, 2, '2025-12-02', 'holiday brownie bites'), (24, 2, '2025-12-03', 'peppermint bark bites'), (25, 2, '2025-12-04', 'holiday jam trio'), (26, 2, '2025-12-05', 'festive notepad'), (27, 2, '2025-12-06', 'scented pine sachet'), (28, 2, '2025-12-07', 'holiday mug'), (29, 2, '2025-12-08', 'shortbread cookie tin'), (30, 2, '2025-12-09', 'dark chocolate assortment'); ``` So, we have two tables: 1. `families` table 2. `deliveries_assigned` table The first table, `family` just has the id and the name of the family. The second table, `deliveries_assigned` has the id, family id, gift date, and gift name. Let's look at the problem statement. ## Problem > Generate a report that returns the dates and families that have no delivery assigned after December 14th, using the `families` and `deliveries_assigned`. > > Each row in the report should be a date and family name that represents the dates in which families don't have a delivery assigned yet. > > Label the columns as `unassigned_date` and `name`. Order the results by `unassigned_date` and `name`, respectively, both in ascending order. Ok, so we need to list the deliveries_assigned for each family first, to check what kind of pattern are we looking for. ```sql SELECT * FROM deliveries_assigned WHERE family_id = 1; ``` ``` sqlite> select * from deliveries_assigned where family_id=1; +----+-----------+------------+---------------------------+ | id | family_id | gift_date | gift_name | +----+-----------+------------+---------------------------+ | 1 | 1 | 2025-12-01 | roasted cashews | | 2 | 1 | 2025-12-02 | cookie decorating kit | | 3 | 1 | 2025-12-03 | dark chocolate assortment | | 4 | 1 | 2025-12-04 | white chocolate candies | | 5 | 1 | 2025-12-05 | reindeer headband | | 6 | 1 | 2025-12-06 | holiday brownie bites | | 7 | 1 | 2025-12-07 | shortbread cookie tin | | 8 | 1 | 2025-12-08 | chocolate chip cookies | | 9 | 1 | 2025-12-11 | holiday jam trio | | 10 | 1 | 2025-12-12 | white chocolate popcorn | | 11 | 1 | 2025-12-14 | holiday jam trio | | 12 | 1 | 2025-12-15 | fudge bites | | 13 | 1 | 2025-12-16 | holiday sticker sheet | | 14 | 1 | 2025-12-18 | hot cocoa bombs | | 15 | 1 | 2025-12-19 | honey roasted nuts | | 16 | 1 | 2025-12-20 | holiday mug | | 17 | 1 | 2025-12-21 | white chocolate candies | | 18 | 1 | 2025-12-22 | puzzle book | | 19 | 1 | 2025-12-23 | snowman plush | | 20 | 1 | 2025-12-24 | scented hand cream | | 21 | 1 | 2025-12-25 | vanilla bean wafers | +----+-----------+------------+---------------------------+ sqlite> ``` So, we are missing gifts for family id `1` on `09`, `10`, `13`, and `17`. But we are only asked for gifts after `14` (December 14th). > Generate a report that returns the dates and families that have **no delivery assigned after December 14th**, using the families and deliveries_assigned. So, we can discard `09`, `10`, and `13` as they are before December 14th. ```sql SELECT * FROM deliveries_assigned WHERE family_id = 1 AND gift_date > '2025-12-14'; ``` It gives the right dates where the gifts are assigned after December 14th. But the problem is we need to get dates which are missing in this `deliveries_assigned` table record for each family. Finding something missing is kind of wired, because you don't have what is missing. Especially for dates, like dates are very painful. We need to find among the sequential order of the dates, when some of the dates are missing, that is simple here, but you can see it could be quite cubersome if we have to manually add each date in the list for comparing with. ### JOINs with NOT EXISTS So, the basic dirty solution is to check the missing dates for each family, one by one. ```sql SELECT families.family_name AS name, dates.column1 AS unassigned_date FROM families JOIN ( VALUES ('2025-12-15'), ('2025-12-16'), ('2025-12-17'), ('2025-12-18'), ('2025-12-19'), ('2025-12-20'), ('2025-12-21'), ('2025-12-22'), ('2025-12-23'), ('2025-12-24'), ('2025-12-25') ) AS dates ON 1=1 WHERE NOT EXISTS ( SELECT 1 FROM deliveries_assigned WHERE deliveries_assigned.family_id = families.id AND deliveries_assigned.gift_date = dates.column1 ) ORDER BY unassigned_date, name; ``` Let's break it down: 1. We have a `families` table with id and family name. 2. We have a `deliveries_assigned` table with id, family id, gift date, and gift name. 3. We create a list of dates from `2025-12-15` to `2025-12-25` using the `VALUES` keyword. - This just appends one date after other and names the columns as `column1` with the table as `dates`. - The `ON` condition is `1=1` to make sure the `WHERE` condition is true for the JOIN to happen. 4. We use the `NOT EXISTS` keyword to check if the `deliveries_assigned` table has a record for each date in the list. - we use `NOT EXISTS` because we want to check if there is no date for each date in the assigned list. 5. We order the results by `unassigned_date` and `name`, respectively, both in ascending order. So, that is not the best way to solve this, I think. ### Recursive CTEs We can generate a table full of dates, and then cross join the table with the `families` table. This will give us all possible combinations of dates and families like the gift should ideally be there for each family from 1st to 25th of December(however we are only interested from 15th to 25th December). Then once we have that full table of combination, we can check from the `deliveries_assigned` table and inner join for each family and filter out the rows which have family_id as `NULL` because some dates will be missing for that family. First we'll create a recursive table of dates from `2025-12-15` to `2025-12-25` ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT * FROM dates; ``` This will give us the dates from `2025-12-15` to `2025-12-25` in a recursive table. What is a recursive table? > A recursive table is a table that is defined as a combination of itself. So, we can create a recursive table of dates from `2025-12-15` to `2025-12-25`, The base case is `2025-12-15`, and the recursive case is `SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25'` which means it will call it recursively for `2025-12-16` on the first call inside it because of `+1 day` as the interval in the `date` function. The date function is a [function](https://sqlite.org/lang_datefunc.html#modifiers) which takes in a date and we can add modifiers to it to manipulate or extract parts of the date. Here we have added the modifier as `+1 day` which will increment the day by one. We then call that in the `dates` CTE again till we have the `date` as less then `2025-12-25`. Till then we will have created all the `dates` from `15` till `25` including `25`. We can just type in the values manually as we did in the first dirty solution, but I wanted to see how we can generate dates dynamically in sqlite. ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT * FROM dates; +------------+ | gift_date | +------------+ | 2025-12-15 | | 2025-12-16 | | 2025-12-17 | | 2025-12-18 | | 2025-12-19 | | 2025-12-20 | | 2025-12-21 | | 2025-12-22 | | 2025-12-23 | | 2025-12-24 | | 2025-12-25 | +------------+ Run Time: real 0.000 user 0.000225 sys 0.000009 sqlite> ``` Now we'll cross join the `families` table with the `dates` table. ```sql SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates; ``` OK! Wait include the `dates` CTE above too, was just simplifying the query. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates; ``` This will give us the full table of combination of dates and families. ```sql SELECT COUNT(*) FROM families; ``` ```sql sqlite> SELECT COUNT(*) FROM families; +----------+ | COUNT(*) | +----------+ | 250 | +----------+ Run Time: real 0.000 user 0.000113 sys 0.000006 sqlite> ``` That is : - There are 11 dates right? `2025-12-15` to `2025-12-25`, -> `15` (1), `16` (2), `17` (3), `18` (4), `19` (5), `20` (6), `21` (7), `22` (8), `23` (9), `24` (10), `25` (11) - There are I think `250` families. - So, a `CROSS JOIN` will give us `11 * 250 = 2750` rows. ```sql SELECT COUNT(*) FROM (WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates) as count; ``` 11 dates for each family. ``` sqlite> SELECT COUNT(*) FROM (WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates) as count; +----------+ | COUNT(*) | +----------+ | 2750 | +----------+ Run Time: real 0.001 user 0.000407 sys 0.000000 sqlite> ``` Now, we simply have to join the `deliveries_assigned` table with the above table. Why? Because we need to map which dates are assigned and which dates are missing. We need to do a which join? LEFT, RIGHT or INNER LEFT WHy? Because, the left table will have all the dates, and the right table will have the assigned dates. **We need all the records from the `left` table (the combination, cross join table)** We need all the rows in the `left` or the combination(cross join) of date and family table as to map which dates are assigned and which dates are missing. If the `left` table has a record, then it means the date is assigned. If the `right` table has a record i.e. the gift is assigned, then it means the date is not assigned. Hence, we can simply then filter out the relations with `NULL` as the `family_id` in the `deliveries_assigned` table. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date ``` ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date LIMIT 30; +-----------+---------------+------------+----+-----------+------------+-------------------------+ | family_id | family_name | gift_date | id | family_id | gift_date | gift_name | +-----------+---------------+------------+----+-----------+------------+-------------------------+ | 1 | Isla Martinez | 2025-12-15 | 12 | 1 | 2025-12-15 | fudge bites | | 1 | Isla Martinez | 2025-12-16 | 13 | 1 | 2025-12-16 | holiday sticker sheet | | 1 | Isla Martinez | 2025-12-17 | | | | | | 1 | Isla Martinez | 2025-12-18 | 14 | 1 | 2025-12-18 | hot cocoa bombs | | 1 | Isla Martinez | 2025-12-19 | 15 | 1 | 2025-12-19 | honey roasted nuts | | 1 | Isla Martinez | 2025-12-20 | 16 | 1 | 2025-12-20 | holiday mug | | 1 | Isla Martinez | 2025-12-21 | 17 | 1 | 2025-12-21 | white chocolate candies | | 1 | Isla Martinez | 2025-12-22 | 18 | 1 | 2025-12-22 | puzzle book | | 1 | Isla Martinez | 2025-12-23 | 19 | 1 | 2025-12-23 | snowman plush | | 1 | Isla Martinez | 2025-12-24 | 20 | 1 | 2025-12-24 | scented hand cream | | 1 | Isla Martinez | 2025-12-25 | 21 | 1 | 2025-12-25 | vanilla bean wafers | | 2 | Nolan Garcia | 2025-12-15 | 36 | 2 | 2025-12-15 | mini marshmallow tubes | | 2 | Nolan Garcia | 2025-12-16 | 37 | 2 | 2025-12-16 | white chocolate candies | | 2 | Nolan Garcia | 2025-12-17 | 38 | 2 | 2025-12-17 | gingerbread cookie kit | | 2 | Nolan Garcia | 2025-12-18 | 39 | 2 | 2025-12-18 | family card game | | 2 | Nolan Garcia | 2025-12-19 | | | | | | 2 | Nolan Garcia | 2025-12-20 | 40 | 2 | 2025-12-20 | santa hat | | 2 | Nolan Garcia | 2025-12-21 | 41 | 2 | 2025-12-21 | holiday sticker sheet | | 2 | Nolan Garcia | 2025-12-22 | | | | | | 2 | Nolan Garcia | 2025-12-23 | 42 | 2 | 2025-12-23 | pecan praline bites | | 2 | Nolan Garcia | 2025-12-24 | | | | | | 2 | Nolan Garcia | 2025-12-25 | 43 | 2 | 2025-12-25 | santa hat | | 3 | Yara Chen | 2025-12-15 | 57 | 3 | 2025-12-15 | peppermint bark bites | | 3 | Yara Chen | 2025-12-16 | | | | | | 3 | Yara Chen | 2025-12-17 | | | | | | 3 | Yara Chen | 2025-12-18 | 58 | 3 | 2025-12-18 | cheddar popcorn | | 3 | Yara Chen | 2025-12-19 | | | | | | 3 | Yara Chen | 2025-12-20 | 59 | 3 | 2025-12-20 | festive notepad | | 3 | Yara Chen | 2025-12-21 | 60 | 3 | 2025-12-21 | fruit assortment | | 3 | Yara Chen | 2025-12-22 | | | | | +-----------+---------------+------------+----+-----------+------------+-------------------------+ ... ... | 249 | Jude Bautista | 2025-12-24 | | | | | | 249 | Jude Bautista | 2025-12-25 | 5073 | 249 | 2025-12-25 | almond brittle | | 250 | Bianca Muller | 2025-12-15 | 5086 | 250 | 2025-12-15 | cocoa mix bundle | | 250 | Bianca Muller | 2025-12-16 | 5087 | 250 | 2025-12-16 | cookie decorating kit | | 250 | Bianca Muller | 2025-12-17 | 5088 | 250 | 2025-12-17 | shortbread cookie tin | | 250 | Bianca Muller | 2025-12-18 | | | | | | 250 | Bianca Muller | 2025-12-19 | 5089 | 250 | 2025-12-19 | snowflake candle | | 250 | Bianca Muller | 2025-12-20 | | | | | | 250 | Bianca Muller | 2025-12-21 | 5090 | 250 | 2025-12-21 | trail mix trio | | 250 | Bianca Muller | 2025-12-22 | 5091 | 250 | 2025-12-22 | shortbread cookie tin | | 250 | Bianca Muller | 2025-12-23 | | | | | | 250 | Bianca Muller | 2025-12-24 | | | | | | 250 | Bianca Muller | 2025-12-25 | 5092 | 250 | 2025-12-25 | gingerbread cookie kit | +-----------+-------------------+------------+------+-----------+------------+----------------------------+ Run Time: real 0.028 user 0.013392 sys 0.013096 sqlite> ``` This will give all the gifts that have been assigned as well as not assigned for each family on each date from `2025-12-15` to `2025-12-25`. Now, you can see, what we need and what we don't we simply can get the gifts or records with the `family_id` in the `deliveries_assigned` table as `NULL`, since there was no record for the family_id in the `deliveries_assigned` table for that date. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ``` So, a simple `WHERE` clause with deliveries_assigned.family_id `IS NULL` will give us the missing dates for each family. ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ...> ; +-----------+-------------------+------------+----+-----------+-----------+-----------+ | family_id | family_name | gift_date | id | family_id | gift_date | gift_name | +-----------+-------------------+------------+----+-----------+-----------+-----------+ | 1 | Isla Martinez | 2025-12-17 | | | | | | 2 | Nolan Garcia | 2025-12-19 | | | | | | 2 | Nolan Garcia | 2025-12-22 | | | | | | 2 | Nolan Garcia | 2025-12-24 | | | | | | 3 | Yara Chen | 2025-12-16 | | | | | | 3 | Yara Chen | 2025-12-17 | | | | | | 3 | Yara Chen | 2025-12-19 | | | | | | 3 | Yara Chen | 2025-12-22 | | | | | | 4 | Tariq Nguyen | 2025-12-16 | | | | | ... ... | 247 | Malik Kim | 2025-12-18 | | | | | | 247 | Malik Kim | 2025-12-20 | | | | | | 247 | Malik Kim | 2025-12-21 | | | | | | 248 | Tariq Flores | 2025-12-16 | | | | | | 248 | Tariq Flores | 2025-12-17 | | | | | | 248 | Tariq Flores | 2025-12-18 | | | | | | 249 | Jude Bautista | 2025-12-16 | | | | | | 249 | Jude Bautista | 2025-12-20 | | | | | | 249 | Jude Bautista | 2025-12-21 | | | | | | 249 | Jude Bautista | 2025-12-23 | | | | | | 249 | Jude Bautista | 2025-12-24 | | | | | | 250 | Bianca Muller | 2025-12-18 | | | | | | 250 | Bianca Muller | 2025-12-20 | | | | | | 250 | Bianca Muller | 2025-12-23 | | | | | | 250 | Bianca Muller | 2025-12-24 | | | | | +-----------+-------------------+------------+----+-----------+-----------+-----------+ Run Time: real 0.014 user 0.011508 sys 0.002046 sqlite> ``` All right, we now need to order by `unassigned_date` and `name` which are the `dates` from the `combination` table and the `family_name` from the `families` table. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT combination.gift_date as unassigned_date, family_name FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ORDER BY unassigned_date, family_name; ``` ``` +-----------------+-------------------+ | unassigned_date | family_name | +-----------------+-------------------+ | 2025-12-15 | Adil Rossi | | 2025-12-15 | Aisha Connor | | 2025-12-15 | Amina Perez | | 2025-12-15 | Amina Wong | | 2025-12-15 | Andre Flores | | 2025-12-15 | Anya Singh | | 2025-12-15 | Arjun Wong | | 2025-12-15 | Bianca Connor | | 2025-12-15 | Caleb Petrov | | 2025-12-15 | Caleb Roberts | | 2025-12-15 | Carmen Carter | | 2025-12-15 | Carmen Garcia | | 2025-12-15 | Casey Flores | | 2025-12-15 | Chi Hughes | | 2025-12-15 | Clara Johnson | | 2025-12-15 | Dara Bautista | | 2025-12-15 | David Ramirez | | 2025-12-15 | Elias Petrov | | 2025-12-15 | Elias Petrov | | 2025-12-15 | Ethan Flores | | 2025-12-15 | Eva Gonzalez | ... ... | 2025-12-25 | Owen Park | | 2025-12-25 | Priya Khan | | 2025-12-25 | Rafael Singh | | 2025-12-25 | Ravi Abdallah | | 2025-12-25 | Ravi Mitchell | | 2025-12-25 | Rosa Turner | | 2025-12-25 | Sara Jensen | | 2025-12-25 | Sara Lopez | | 2025-12-25 | Sara Rossi | | 2025-12-25 | Sarah Phillips | | 2025-12-25 | Seth Garcia | | 2025-12-25 | Sienna Lopez | | 2025-12-25 | Sofia Nakamura | | 2025-12-25 | Tariq Nguyen | | 2025-12-25 | Uma Ali | | 2025-12-25 | Uma Phillips | | 2025-12-25 | Yara Chen | | 2025-12-25 | Yara Rossi | | 2025-12-25 | Yusuf Ali | | 2025-12-25 | Yusuf Hansen | | 2025-12-25 | Yusuf Perez | | 2025-12-25 | Yusuf Rossi | | 2025-12-25 | Zara Khan | +-----------------+-------------------+ Run Time: real 0.008 user 0.004613 sys 0.002845 sqlite> ``` Phew! Ok, that looks a mamoth query. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT combination.gift_date as unassigned_date, family_name FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ORDER BY unassigned_date, family_name; ``` So, to recap - Generate a table of dates from `2025-12-15` to `2025-12-25` using a recursive CTE. - Create a table of combinations of `families` and `dates` using a cross join. - Left join the `deliveries_assigned` table with the combination table. - Filter out the rows where the `family_id` in the `deliveries_assigned` table is `NULL`. - Order the results by `unassigned_date` and `name`, respectively, both in ascending order. Simple right? That's it from day 6. It getting serious out there!

Advent of SQL 2025 Day 5: EchoTrack Wrapped

Sat, 20 Dec 2025 00:00:00 UTC

## Advent of SQL Day 5 - EchoTrack Wrapped It is day 5 of advent of SQL. Let's get rollin. It looks like a good problem. I am excited! Here's the SQL to get started. ```sql DROP TABLE IF EXISTS listening_logs; CREATE TABLE listening_logs ( id INTEGER PRIMARY KEY, user_name TEXT, artist TEXT, played_at TIMESTAMP, content_type TEXT ); INSERT INTO listening_logs (id, user_name, artist, played_at, content_type) VALUES (1, 'Zoe Garcia', 'Arijit Singh', '2025-04-08 00:21:53', 'song'), (2, 'Zoe Garcia', 'Huberman Lab', '2025-11-10 19:18:47', 'podcast'), (3, 'Zoe Garcia', 'Huberman Lab', '2025-01-20 15:31:02', 'podcast'), (4, 'Zoe Garcia', 'Arijit Singh', '2025-01-06 17:33:11', 'song'), (5, 'Zoe Garcia', 'Candace', '2025-03-06 14:07:54', 'podcast'), (6, 'Zoe Garcia', 'Arijit Singh', '2025-06-05 17:57:59', 'song'), (7, 'Zoe Garcia', 'Huberman Lab', '2025-01-01 20:05:22', 'podcast'), (8, 'Zoe Garcia', 'Huberman Lab', '2025-11-01 12:04:03', 'podcast'), (9, 'Zoe Garcia', 'Arijit Singh', '2025-09-28 12:42:12', 'song'), (10, 'Zoe Garcia', 'The Ben Shapiro Show', '2025-09-15 01:05:15', 'podcast'), (11, 'Zoe Garcia', 'Arijit Singh', '2025-04-26 05:31:02', 'song'), (12, 'Zoe Garcia', 'Arijit Singh', '2025-10-13 17:34:03', 'song'), (13, 'Zoe Garcia', 'Mariah Carey', '2025-01-20 11:21:37', 'song'), (14, 'Zoe Garcia', 'Arijit Singh', '2025-11-28 03:55:31', 'song'), (15, 'Zoe Garcia', 'Arijit Singh', '2025-07-17 05:18:16', 'song'), (16, 'Zoe Garcia', 'Arijit Singh', '2025-08-20 02:07:45', 'song'), (17, 'Zoe Garcia', 'Kendrick Lamar', '2025-02-16 13:25:27', 'song'), (18, 'Zoe Garcia', 'Huberman Lab', '2025-08-13 19:55:00', 'podcast'), (19, 'Zoe Garcia', 'Bruno Mars', '2025-09-13 07:09:43', 'song'), (20, 'Zoe Garcia', 'Arijit Singh', '2025-04-12 06:30:44', 'song'); ``` Let's open a SQLite shell and get started. ``` $ sqlite3 SQLite version 3.50.4 2025-07-30 19:33:53 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day5-inserts.sql sqlite> .schema CREATE TABLE listening_logs ( id INTEGER PRIMARY KEY, user_name TEXT, artist TEXT, played_at TIMESTAMP, content_type TEXT ); sqlite> .mode table sqlite> SELECT * FROM listening_logs LIMIT 20; +----+------------+----------------------+---------------------+--------------+ | id | user_name | artist | played_at | content_type | +----+------------+----------------------+---------------------+--------------+ | 1 | Zoe Garcia | Arijit Singh | 2025-04-08 00:21:53 | song | | 2 | Zoe Garcia | Huberman Lab | 2025-11-10 19:18:47 | podcast | | 3 | Zoe Garcia | Huberman Lab | 2025-01-20 15:31:02 | podcast | | 4 | Zoe Garcia | Arijit Singh | 2025-01-06 17:33:11 | song | | 5 | Zoe Garcia | Candace | 2025-03-06 14:07:54 | podcast | | 6 | Zoe Garcia | Arijit Singh | 2025-06-05 17:57:59 | song | | 7 | Zoe Garcia | Huberman Lab | 2025-01-01 20:05:22 | podcast | | 8 | Zoe Garcia | Huberman Lab | 2025-11-01 12:04:03 | podcast | | 9 | Zoe Garcia | Arijit Singh | 2025-09-28 12:42:12 | song | | 10 | Zoe Garcia | The Ben Shapiro Show | 2025-09-15 01:05:15 | podcast | | 11 | Zoe Garcia | Arijit Singh | 2025-04-26 05:31:02 | song | | 12 | Zoe Garcia | Arijit Singh | 2025-10-13 17:34:03 | song | | 13 | Zoe Garcia | Mariah Carey | 2025-01-20 11:21:37 | song | | 14 | Zoe Garcia | Arijit Singh | 2025-11-28 03:55:31 | song | | 15 | Zoe Garcia | Arijit Singh | 2025-07-17 05:18:16 | song | | 16 | Zoe Garcia | Arijit Singh | 2025-08-20 02:07:45 | song | | 17 | Zoe Garcia | Kendrick Lamar | 2025-02-16 13:25:27 | song | | 18 | Zoe Garcia | Huberman Lab | 2025-08-13 19:55:00 | podcast | | 19 | Zoe Garcia | Bruno Mars | 2025-09-13 07:09:43 | song | | 20 | Zoe Garcia | Arijit Singh | 2025-04-12 06:30:44 | song | +----+------------+----------------------+---------------------+--------------+ sqlite> sqlite> SELECT COUNT(*) FROM listening_logs; +----------+ | COUNT(*) | +----------+ | 18174 | +----------+ sqlite> ``` OK! We have around 18k records in a single table! That's a lot but not not much! Let's see what we have to do ## Problem > Write a query that returns the top 3 artists per user. Order the results by the most played Alright, this is quite a problem to solve, if you are thinking, it easy peasy, then hold on! We clearly see, that we have 2 columns of our interest. 1. `user_name` 2. `artist` We need to group for each user his most played artists, and we need to rank those top 3 artist per user. And each entry is a song or podcast that the user has litened to. We need to aggregate, group, and then rank, and then what? How would you chunk out the top three? It's time to put your SQL glasses and gloves on, it's getting colder! ### Counting Artists Per User Let's take one step at a time, we need to first count how many times each artist has been played for that user. ```sql SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` OK, simple right? Select all the usernames, artist and then group by the username and the artist and count the number of times the user had played that artist. Simply then order by username, the playcount and the artist (if there is a tie in count, I think we can choose the artist name as the breaker) But it gives all the artists, not just the top 3, it ranks them in the decreasing order of the number of plays, but we only want to list the top 3 per user. That's tricky! ### With SELF JOIN What if we can join the table with itself? Then we can compare the number of times the user has played the artist, with the number of times the user has played another artist. Then we can remove the artist if it has played less than 3 times, this then will filter out the top 3 for us. ```sql SELECT a.user_name, a.artist AS current_artist, a.play_count AS current_plays, b.artist AS other_artist, b.play_count AS other_plays FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name ORDER BY a.user_name, a.play_count DESC, b.play_count DESC; ``` This would create a cross join of sorts between the same table. - Select the required columns (user_name, artist and count) Look at this part This is `a` ```sql -- a SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist ``` Then we need to join this with itself - Select the required columns from the same table (user_name, artist and count) This is `b` ```sql -- b SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist ``` Both `a` and `b` are the same, just that we want a cross join of sorts. And then we need to join `a` and `b` - Join with itself on the user_name - Order by user_name, play_count desc, artist ```sql SELECT a.user_name, a.artist AS current_artist, a.play_count AS current_plays, b.artist AS other_artist, b.play_count AS other_plays FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name ORDER BY a.user_name, a.play_count DESC, b.play_count DESC; ``` Now, we want to keep the rows where the `b` table has play count greater than `a` table, or if they are equal, then if the `b` table has artist less than `a` table. To do that we can continue the `JOIN` condition with `AND` and add `b.play_count > a.play_count` and `b.artist < a.artist` in case of a tie. The idea here is subtle: - For a given artist `a`, we count how many artists `b` (for the same user) have more plays, or the same plays but come earlier alphabetically. - If fewer than 3 artists beat `a`, then `a` must be in the top 3. So the query becomes this: ```sql SELECT a.user_name, a.artist, a.play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name AND ( b.play_count > a.play_count OR (b.play_count = a.play_count AND b.artist < a.artist) ) GROUP BY a.user_name, a.artist, a.play_count HAVING COUNT(b.artist) < 3 ORDER BY a.user_name, a.play_count DESC, a.artist; ``` ``` sqlite> SELECT a.user_name, a.artist, a.play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name AND ( b.play_count > a.play_count OR (b.play_count = a.play_count AND b.artist < a.artist) ) GROUP BY a.user_name, a.artist, a.play_count HAVING COUNT(b.artist) < 3 ORDER BY a.user_name, a.play_count DESC, a.artist; +-------------------+--------------------------------------------+------------+ | user_name | artist | play_count | +-------------------+--------------------------------------------+------------+ | Abigail Hernandez | Ed Sheeran | 78 | | Abigail Hernandez | Rotten Mango | 15 | | Abigail Hernandez | Billie Eilish | 4 | | Adrian Cox | Kendrick Lamar | 128 | | Adrian Cox | Stuff You Should Know | 30 | | Adrian Cox | Fuerza Regida | 6 | | Alex Rivera | Ed Sheeran | 274 | | Alex Rivera | Call Her Daddy (Alex Cooper) | 42 | | Alex Rivera | Green Day | 11 | | Anders Nilsson | Snow Patrol | 101 | | Anders Nilsson | SmartLess | 29 | | Anders Nilsson | Blink-182 | 5 | | Anthony King | Pentatonix | 114 | | Anthony King | The Tucker Carlson Show | 14 | | Anthony King | Angels & Airwaves | 5 | ... ... | Zara Sheikh | Green Day | 138 | | Zara Sheikh | This Past Weekend w Theo Von | 20 | | Zara Sheikh | The Beatles | 7 | | Zoe Garcia | Arijit Singh | 50 | | Zoe Garcia | Huberman Lab | 14 | | Zoe Garcia | Kendrick Lamar | 5 | | Zoe Wilson | Pentatonix | 65 | | Zoe Wilson | The Mel Robbins Podcast | 14 | | Zoe Wilson | Angels & Airwaves | 4 | | Zuri Okafor | Kendrick Lamar | 96 | | Zuri Okafor | The Tim Dillon Show | 14 | | Zuri Okafor | Ed Sheeran | 5 | +-------------------+--------------------------------------------+------------+ Run Time: real 0.088 user 0.082510 sys 0.003999 ``` This is the final query, it looks long, it might be not the best way to do it, but its definitely not the worst way to do it. ### With Window Functions Ok! I don't know window functions, but I searched and found that we could partition things before we group them or order them in the final result set. That's what we want right? We had grouped the logs for each username and artist and counted the number of plays. Now we want to rank the artists for each user in the decreasing order of number of plays. So, we start with the same thing: ```sql SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` You can see that we have `user_name` column, what if we could separate out the users, and then we could rank the artists for each user separately. For that we can use `ROW_NUMBER` window function. This function needs a `PARTITION BY` clause which lets us create separate partitions based on certain columns, and then we can rank them using the `ORDER BY` clause as we do with ordinary statement. That becomes the row_number, which we can use as a rank to rank each artist for a user based on the number of time the user has played him/her. ```sql SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ``` Here, `ROW_NUMBER` is a window function that assigns a rank to each row in the result set. It partitions the result set by `user_name` and orders the rows in the decreasing order of `COUNT(*)` and `artist` name. So, imagine this table: ``` +-------------------+-------------------------------------+------------+-------+ | user_name | artist | play_count | ranks | +-------------------+-------------------------------------+------------+-------+ | Abigail Hernandez | Ed Sheeran | 78 | 1 | | Abigail Hernandez | Rotten Mango | 15 | 2 | | Abigail Hernandez | Billie Eilish | 4 | 3 | | Abigail Hernandez | Hans Zimmer | 4 | 4 | | Abigail Hernandez | John Legend | 3 | 5 | | Abigail Hernandez | John Williams | 3 | 6 | | Abigail Hernandez | The Beatles | 3 | 7 | | Abigail Hernandez | The Rolling Stones | 3 | 8 | | Abigail Hernandez | Angels & Airwaves | 2 | 9 | | Abigail Hernandez | Bad Bunny | 2 | 10 | | Abigail Hernandez | Beyonce | 2 | 11 | | Abigail Hernandez | Coldplay | 2 | 12 | | Abigail Hernandez | Foo Fighters | 2 | 13 | | Abigail Hernandez | Fuerza Regida | 2 | 14 | | Abigail Hernandez | Kendrick Lamar | 2 | 15 | | Abigail Hernandez | Ludovico Einaudi | 2 | 16 | | Abigail Hernandez | Mariah Carey | 2 | 17 | | Abigail Hernandez | Pentatonix | 2 | 18 | | Abigail Hernandez | SmartLess | 2 | 19 | | Abigail Hernandez | The Weeknd | 2 | 20 | | Abigail Hernandez | Adele | 1 | 21 | | Abigail Hernandez | Ariana Grande | 1 | 22 | | Abigail Hernandez | Armchair Expert With Dax Shepard | 1 | 23 | | Abigail Hernandez | Bruno Mars | 1 | 24 | | Abigail Hernandez | Candace | 1 | 25 | | Abigail Hernandez | Crime, Conspiracy, Cults and Murder | 1 | 26 | | Abigail Hernandez | Green Day | 1 | 27 | | Abigail Hernandez | Matt and Shanes Secret Podcast | 1 | 28 | | Abigail Hernandez | On Purpose With Jay Shetty | 1 | 29 | | Abigail Hernandez | Snow Patrol | 1 | 30 | | Abigail Hernandez | Sufjan Stevens | 1 | 31 | | Abigail Hernandez | Taylor Swift | 1 | 32 | | Abigail Hernandez | The Mel Robbins Podcast | 1 | 33 | | Abigail Hernandez | Unseen | 1 | 34 | | Adrian Cox | Kendrick Lamar | 128 | 1 | | Adrian Cox | Stuff You Should Know | 30 | 2 | | Adrian Cox | Fuerza Regida | 6 | 3 | | Adrian Cox | Pentatonix | 6 | 4 | | Adrian Cox | Taylor Swift | 6 | 5 | | Adrian Cox | Snow Patrol | 5 | 6 | +-------------------+-------------------------------------+------------+-------+ ``` Now, we have ranked the artist for each user, there are `34` artist played by `Abigail Hernandez` so there are `34` ranks. Now, we need to filter out the top 3 artist for each user. That would be simple right? Just add the `WHERE` clause: ```sql SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs WHERE ranks <= 3 GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` Well not really! ``` sqlite> SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) FROM listening_logs WHERE ranks <= 3 GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; Run Time: real 0.000 user 0.000110 sys 0.000000 Parse error: misuse of aliased window function ranks ``` We can't filter the window function column with a `WHERE` clause. At the time the `WHERE` clause is evaluated, the `SELECT` list (including ranks) has not been computed yet. So ranks doesn't exist yet! So, now what, so close yet so far! Let's wrap the `SELECT` in a subquery: ```sql SELECT user_name, artist, play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ) ranked WHERE ranks <= 3 ORDER BY user_name, play_count DESC, artist; ``` And there we have it! We wrapped the `SELECT` in a subquery, and now we can filter the window function column with a `WHERE` clause. We can even do it with a `CTE` i.e. common table expression. That's just a subquery with a name. Not sure when it could be handy, but here it is. ```sql WITH ranked AS ( SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ) SELECT user_name, artist, play_count FROM ranked WHERE ranks <= 3 ORDER BY user_name, play_count DESC, artist; ``` So, that is day 5, ok that is getting a little tricky now! Off to day 6.

Advent of SQL 2025 Day 4: WinterFest Volunteers

Fri, 19 Dec 2025 00:00:00 UTC

## Advent of SQL Day 4 WinterFest Volunteers It is day 4 of advent of SQL. No fuss, straight to the problem, the elves and humans are getting dumber as the days progress. Let's download the SQL inserts for the day. And load it into a SQLite shell. ```sql DROP TABLE IF EXISTS official_shifts; DROP TABLE IF EXISTS last_minute_signups; CREATE TABLE official_shifts ( id INT PRIMARY KEY, volunteer_name TEXT, role TEXT, shift_time TEXT, age_group TEXT, code TEXT ); CREATE TABLE last_minute_signups ( id INT PRIMARY KEY, volunteer_name TEXT, assigned_task TEXT, time_slot TEXT ); INSERT INTO official_shifts (id, volunteer_name, role, shift_time, age_group, code) VALUES (1, 'Jude Thompson', 'choir_assistant', '12:00 PM', 'senior', NULL), (2, 'Mateo Cruz', 'choir_assistant', '12:00 PM', 'senior', NULL), (3, 'Olivia Dubois', 'choir_assistant', '2:00 PM', 'teen', 'A1'), (4, 'Jeff Bezos', 'choir_assistant', '10:00 AM', 'adult', 'X7'), (5, 'Kian Rahimi', 'stage_setup', '12:00 PM', 'adult', 'X7'), (6, 'Haruto Sato', 'cocoa_station', '10:00 AM', 'adult', 'X7'), (7, 'Uma Singh', 'parking_support', '10:00 AM', 'adult', NULL), (8, 'Owen Scott', 'parking_support', '10:00 AM', 'adult', 'X7'), (9, 'Adil Rahman', 'stage_setup', '2:00 PM', 'adult', 'A1'), (10, 'Aaron Diaz', 'choir_assistant', '2:00 PM', 'senior', 'X7'), (11, 'Carter Lewis', 'cocoa_station', '10:00 AM', 'senior', 'B2'), (12, 'Anya Pavlov', 'stage_setup', '10:00 AM', 'senior', 'OLD'), (13, 'Ethan Brown', 'stage_setup', '2:00 PM', 'adult', 'A1'), (14, 'Lucia Fernandez', 'choir_assistant', '12:00 PM', 'senior', 'X7'), (15, 'Casey Morgan', 'choir_assistant', '12:00 PM', 'teen', 'OLD'); INSERT INTO last_minute_signups (id, volunteer_name, assigned_task, time_slot) VALUES (1, 'Jude Thompson', 'Choir', 'noon'), (2, 'Mateo Cruz', 'choir', 'noon'), (3, 'Olivia Dubois', 'choir', '2 PM'), (4, 'Jeff Bezos', 'choir assistant', '10AM'), (5, 'Kian Rahimi', 'stage setup', 'noon'), (6, 'Haruto Sato', 'cocoa station', '10AM'), (7, 'Uma Singh', 'parking_support', '10AM'), (8, 'Owen Scott', 'parking', '10AM'), (9, 'Adil Rahman', 'Stage-Setup', '2 PM'), (10, 'Aaron Diaz', 'Choir', '2 PM'), (11, 'Carter Lewis', 'Cocoa Station', '10AM'), (12, 'Anya Pavlov', 'stage_setup', '10AM'), (13, 'Olivia Brown', 'stage setup', '2 PM'), (14, 'Lena Fischer', 'cocoa station', '2 pm'), (15, 'Nolan Murphy', 'parking-support', '10AM'); ``` Once the data is loaded, let's sneak peak. ```sql SELECT * FROM official_shifts LIMIT 15; ``` ```sql SELECT * FROM last_minute_signups LIMIT 15; ``` Let's count how many rows in each table we have: ```sql SELECT COUNT(*) FROM official_shifts LIMIT 15; SELECT COUNT(*) FROM last_minute_signups LIMIT 15; ``` Alright, the data is visible and we can head on to the problem statement. ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day4_inserts.sql sqlite> .schema CREATE TABLE official_shifts ( id INT PRIMARY KEY, volunteer_name TEXT, role TEXT, shift_time TEXT, age_group TEXT, code TEXT ); CREATE TABLE last_minute_signups ( id INT PRIMARY KEY, volunteer_name TEXT, assigned_task TEXT, time_slot TEXT ); sqlite> SELECT * FROM official_shifts LIMIT 15; 1|Jude Thompson|choir_assistant|12:00 PM|senior| 2|Mateo Cruz|choir_assistant|12:00 PM|senior| 3|Olivia Dubois|choir_assistant|2:00 PM|teen|A1 4|Jeff Bezos|choir_assistant|10:00 AM|adult|X7 5|Kian Rahimi|stage_setup|12:00 PM|adult|X7 6|Haruto Sato|cocoa_station|10:00 AM|adult|X7 7|Uma Singh|parking_support|10:00 AM|adult| 8|Owen Scott|parking_support|10:00 AM|adult|X7 9|Adil Rahman|stage_setup|2:00 PM|adult|A1 10|Aaron Diaz|choir_assistant|2:00 PM|senior|X7 11|Carter Lewis|cocoa_station|10:00 AM|senior|B2 12|Anya Pavlov|stage_setup|10:00 AM|senior|OLD 13|Ethan Brown|stage_setup|2:00 PM|adult|A1 14|Lucia Fernandez|choir_assistant|12:00 PM|senior|X7 15|Casey Morgan|choir_assistant|12:00 PM|teen|OLD sqlite> .mode table sqlite> SELECT * FROM official_shifts LIMIT 15; +----+-----------------+-----------------+------------+-----------+------+ | id | volunteer_name | role | shift_time | age_group | code | +----+-----------------+-----------------+------------+-----------+------+ | 1 | Jude Thompson | choir_assistant | 12:00 PM | senior | | | 2 | Mateo Cruz | choir_assistant | 12:00 PM | senior | | | 3 | Olivia Dubois | choir_assistant | 2:00 PM | teen | A1 | | 4 | Jeff Bezos | choir_assistant | 10:00 AM | adult | X7 | | 5 | Kian Rahimi | stage_setup | 12:00 PM | adult | X7 | | 6 | Haruto Sato | cocoa_station | 10:00 AM | adult | X7 | | 7 | Uma Singh | parking_support | 10:00 AM | adult | | | 8 | Owen Scott | parking_support | 10:00 AM | adult | X7 | | 9 | Adil Rahman | stage_setup | 2:00 PM | adult | A1 | | 10 | Aaron Diaz | choir_assistant | 2:00 PM | senior | X7 | | 11 | Carter Lewis | cocoa_station | 10:00 AM | senior | B2 | | 12 | Anya Pavlov | stage_setup | 10:00 AM | senior | OLD | | 13 | Ethan Brown | stage_setup | 2:00 PM | adult | A1 | | 14 | Lucia Fernandez | choir_assistant | 12:00 PM | senior | X7 | | 15 | Casey Morgan | choir_assistant | 12:00 PM | teen | OLD | +----+-----------------+-----------------+------------+-----------+------+ sqlite> SELECT * FROM last_minute_signups LIMIT 15; +----+----------------+-----------------+-----------+ | id | volunteer_name | assigned_task | time_slot | +----+----------------+-----------------+-----------+ | 1 | Jude Thompson | Choir | noon | | 2 | Mateo Cruz | choir | noon | | 3 | Olivia Dubois | choir | 2 PM | | 4 | Jeff Bezos | choir assistant | 10AM | | 5 | Kian Rahimi | stage setup | noon | | 6 | Haruto Sato | cocoa station | 10AM | | 7 | Uma Singh | parking_support | 10AM | | 8 | Owen Scott | parking | 10AM | | 9 | Adil Rahman | Stage-Setup | 2 PM | | 10 | Aaron Diaz | Choir | 2 PM | | 11 | Carter Lewis | Cocoa Station | 10AM | | 12 | Anya Pavlov | stage_setup | 10AM | | 13 | Olivia Brown | stage setup | 2 PM | | 14 | Lena Fischer | cocoa station | 2 pm | | 15 | Nolan Murphy | parking-support | 10AM | +----+----------------+-----------------+-----------+ sqlite> SELECT count(*) FROM official_shifts; +----------+ | count(*) | +----------+ | 250 | +----------+ sqlite> SELECT count(*) FROM last_minute_signups; +----------+ | count(*) | +----------+ | 126 | +----------+ sqlite> ``` ## Problem Here's the challenge for day 4 > Using the official_shifts and last_minute_signups tables, create a combined de-duplicated volunteer list. > > Ensure the list has standardized role labels of Stage Setup, Cocoa Station, Parking Support, Choir Assistant, Snow Shoveling, Handwarmer Handout. > > Make sure that the timeslot formats follow John's official shifts format. What we have here is a official shift table which could have been entered by the system. However the `last_minute_shift` is messy and has been added from a sheet, so we need to clean it up and combine those two tables data into a single de-duplicated list of volunteers. Let's see what we got ```sql SELECT * FROM last_minute_signups; ``` Ok, we have 126 records and the columns are: 1. `volunteer_name` which doesn't look bad, 2. `assigned_task` which looks wonky 3. `time_slot` is just wild, we have wired definition of times there. And let's look at the `official_shifts` ```sql SELECT * FROM offical_shifts; ``` This looks neat and tidy, nothing looking off from each other. So we need to make sure we are cleaning up the `last_minute_signups` before we merge them. ```sql SELECT DISTINCT assigned_task FROM last_minute_signups; ``` Ok, so casing is one thing we can see, `-` and ` ` space are the things to normalize, and then some inconsistent naming convention like `choir` and `choir assistant`, then `parking_support` and `parking`. We need to clean'em up. We can search for ```sql SELECT id, volunteer_name, time_slot, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task FROM last_minute_signups; ``` We just do a case match for: - `LIKE 'choir%'` which will match any case (`Choir`, `choir`) and also anything after `choir....` like `choir assistant` - `LIKE 'stage%'` which will match any case (`Stage`, `stage`) and also anything after `stage...` like `Stage-Setup`, `stage setup` or `stage_setup`. - `LIKE '%cocoa'` which will match any case (`Cocoa`, `cocoa`) and also anything before or after `...cocoa...` like `Cocoa Station`, `cocoa station`, etc. - `LIKE 'parking%'` which will match any case (`Parking`, `parking`) and also anything after `parking...` like `parking-support` or `parking_support`, etc. - `LIKE 'hand%'` which will match any case (`Hand`, `hand`) and also anything after `hand...` like `handwarmer handout`, `handwarmers`, `Handwarmer-Handout`, etc. - `LIKE '%shovel%'` which will match any case (`Shovel`, `shovel`) and also anything before and after `...shovel...` like `Snow-Shoveling`, `shovel`, `snow shoveling`, etc. Ok, now this looks unified for the `assigned_task`. ``` sqlite> SELECT id, volunteer_name, time_slot, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task FROM last_minute_signups; +-----+-------------------+-----------+-----------------+ | id | volunteer_name | time_slot | assigned_task | +-----+-------------------+-----------+-----------------+ | 1 | Jude Thompson | noon | choir_assistant | | 2 | Mateo Cruz | noon | choir_assistant | | 3 | Olivia Dubois | 2 PM | choir_assistant | | 4 | Jeff Bezos | 10AM | choir_assistant | | 5 | Kian Rahimi | noon | stage_setup | | 6 | Haruto Sato | 10AM | cocoa_station | | 7 | Uma Singh | 10AM | parking_support | | 8 | Owen Scott | 10AM | parking_support | | 9 | Adil Rahman | 2 PM | stage_setup | | 10 | Aaron Diaz | 2 PM | choir_assistant | | 11 | Carter Lewis | 10AM | cocoa_station | | 12 | Anya Pavlov | 10AM | stage_setup | | 13 | Olivia Brown | 2 PM | stage_setup | | 14 | Lena Fischer | 2 pm | cocoa_station | | 15 | Nolan Murphy | 10AM | parking_support | +-----+-------------------+-----------+-----------------+ ``` We need to make it for the time slot too. ```sql SELECT DISTINCT time_slot FROM last_minute_signups ``` ``` sqlite> SELECT DISTINCT time_slot FROM last_minute_signups; +-----------+ | time_slot | +-----------+ | noon | | 2 PM | | 10AM | | 2 pm | | 10 am | +-----------+ sqlite> ``` Since we have to follow the `official_shifts` let's check over there. ```sql SELECT distinct shift_time FROM official_shifts; ``` ``` sqlite> SELECT distinct shift_time FROM official_shifts; +------------+ | shift_time | +------------+ | 12:00 PM | | 2:00 PM | | 10:00 AM | +------------+ ``` Ok we have only 3 times to change. Alright, we have some small things to standardize. ```sql SELECT DISTINCT CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups; ``` ``` sqlite> SELECT DISTINCT CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' END AS time_slot FROM last_minute_signups; +-----------+ | time_slot | +-----------+ | 12:00 PM | | 2:00 PM | | 10:00 AM | +-----------+ sqlite> ``` So, we have simply standardize the time_slots. - `LIKE '2%'` will match any case but we need `LIKE` to match the `%` rest of the stuff after `2`. - `LIKE '10%'` will match any case but we need `LIKE` to match the `%` rest of the stuff after `10`. We can't keep it `LIKE '1%'` as it will match `1:00` as well - `LIKE 'noon'` will match any case of `noon` like `NOON` or `Noon`, etc. And we need to cast it to the `HH:MM AM or PM` format. So, now we can combine them. ```sql SELECT id, volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 PM' END AS shift_time FROM last_minute_signups; ``` Just changed the column names from `assigned_task` to `role` and `time_slot` to `shift_time` as per the name convention in the `official_shifts` table. Phew! its a long statement. ``` sqlite> SELECT id, volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups; +-----+-------------------+-----------------+-----------+ | id | volunteer_name | assigned_task | time_slot | +-----+-------------------+-----------------+-----------+ | 1 | Jude Thompson | choir_assistant | 12:00 PM | | 2 | Mateo Cruz | choir_assistant | 12:00 PM | | 3 | Olivia Dubois | choir_assistant | 2:00 PM | | 4 | Jeff Bezos | choir_assistant | 10:00 AM | | 5 | Kian Rahimi | stage_setup | 12:00 PM | | 6 | Haruto Sato | cocoa_station | 10:00 AM | | 7 | Uma Singh | parking_support | 10:00 AM | | 8 | Owen Scott | parking_support | 10:00 AM | | 9 | Adil Rahman | stage_setup | 2:00 PM | | 10 | Aaron Diaz | choir_assistant | 2:00 PM | | 11 | Carter Lewis | cocoa_station | 10:00 AM | | 12 | Anya Pavlov | stage_setup | 10:00 AM | | 13 | Olivia Brown | stage_setup | 2:00 PM | | 14 | Lena Fischer | cocoa_station | 2:00 PM | | 15 | Nolan Murphy | parking_support | 10:00 AM | +-----+-------------------+-----------------+-----------+ ``` SO, now we have the table of `last_minute_signups` cleaned up, just with select, we can update them if needed. We now need to combine the both tables, cleaned up `last_minute_signups` and the `official_shifts`, we can use `UNION` to take out the duplicates from the two selection. REMEMBER to order the rows correctly in both the tables. - volunteer_name - role - shift_time I don't think name should be same, but I am keeping it same for clarity. Why `UNION` - Because we have data in both the tables. - We don't have a relation in both of the tables, those are the same tables just that the columns are not cleaned or in proper format. - We want to grab all of them from one, all from other table, and remove the duplicates, that's a definition of `UNION` We can't use `UNION ALL` as it would include all the rows from both the tables without removing duplicates. ```sql SELECT volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups UNION SELECT volunteer_name, role, shift_time FROM official_shifts ORDER BY volunteer_name; ``` Ok, that is a mess, isn't it? ``` sqlite> SELECT volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups UNION SELECT volunteer_name, role, shift_time FROM official_shifts ORDER BY volunteer_name; +-------------------+-----------------+-----------+ | volunteer_name | assigned_task | time_slot | +-------------------+-----------------+-----------+ | Aaron Carter | parking_support | 2:00 PM | | Aaron Diaz | choir_assistant | 2:00 PM | | Aaron Diaz | choir_assistant | 2:00 PM | | Aaron Evans | cocoa_station | 2:00 PM | | Aaron Francis | hand_warmer | 2:00 PM | | Abigail Hernandez | stage_setup | 10:00 AM | | Adam King | stage_setup | 10:00 AM | | Adil Foster | stage_setup | 2:00 PM | | Adil Rahman | stage_setup | 2:00 PM | | Adil Rahman | stage_setup | 2:00 PM | | Adrian Cox | cocoa_station | 10:00 AM | | Aisha Bennett | cocoa_station | 12:00 PM | | Aisha Khan | choir_assistant | 12:00 PM | | Aisha Khan | choir_assistant | 12:00 PM | | Aisha Mohammed | cocoa_station | 2:00 PM | +-------------------+-----------------+-----------+ ``` There we have it. Let's count the number of distinct volunteers in the shifts. ```sql SELECT COUNT(*) FROM (SELECT volunteer_name, role, shift_time FROM official_shifts UNION SELECT volunteer_name, CASE WHEN assigned_task LIKE '%choir%' THEN 'choir_assistant' WHEN assigned_task LIKE '%stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE '%parking%' THEN 'parking_support' WHEN assigned_task LIKE '%hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS shift_time FROM last_minute_signups) AS volunteers; ``` Just counted the full union of the statement using `SELECT COUNT(*) FROM AS volunteers) ``` sqlite> SELECT COUNT(*) FROM (SELECT volunteer_name, role, shift_time FROM official_shifts UNION SELECT volunteer_name, CASE WHEN assigned_task LIKE '%choir%' THEN 'choir_assistant' WHEN assigned_task LIKE '%stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE '%parking%' THEN 'parking_support' WHEN assigned_task LIKE '%hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS shift_time FROM last_minute_signups) AS volunteers; +------------+ | volunteers | +------------+ | 284 | +------------+ sqlite> ``` So, we have around `284` rows. Looks good. Pinebrook can see the volunteer list now. The cleaned one. Off to day 5!

Advent of SQL 2025 Day 3: Hotline Messages

Thu, 18 Dec 2025 00:00:00 UTC

## Advent of SQL Day 3 - Hotline Messages This is day 3 from the Advent of SQL ## Grab the SQL Statements Let's take the insert statements i.e. to create and populate tables and rows into the database. I am using SQLite. It works without any special shenanigans, as it was intended to used for Postgres, but the table and use case looks very simple, so nothing specific to Postgres used yet! We are good! Here is the SQL setup, if you want to play it in the playground here: ```sql DROP TABLE IF EXISTS hotline_messages; CREATE TABLE hotline_messages ( id INT PRIMARY KEY, caller_name TEXT, transcript TEXT, tag TEXT, status TEXT ); INSERT INTO hotline_messages (id, caller_name, transcript, tag, status) VALUES (1, 'Saanvi A.', 'I just found a refrigerator portal that leads to a disco party hosted by dancing llamas—please send help!', 'possible dragon', 'spam'), (2, 'Fatima Q.', 'Hi Santa, I would love a magical unicorn that lights up at night!', 'wish list', NULL), (3, 'Lillian Z.', 'Hi Santa, I would love the magical fairy garden set, please!', 'wish list', 'approved'), (4, 'Carter Y.', 'Thank you, Santa, for making Christmas so special with your wonderful spirit!', 'thank you', 'approved'), (5, 'Omar R.', 'Hi Santa, I would love a rainbow unicorn plushie that has a glittery horn!', 'wish list', 'approved'), (6, 'Diego Y.', 'Hi Santa, I would love a magical unicorn plushie that glows in the dark!', 'wish list', NULL), (7, 'Layla X.', 'Thank you, Santa, for spreading joy and magic every Christmas!', 'thank you', NULL), (8, 'Sophia K.', 'Santa, my cat said she wants to visit the candy cane forest next week.', NULL, NULL), (9, 'Eli H.', 'Hi Santa, I would love the magical fairy castle with twinkling lights!', 'wish list', 'approved'), (10, 'Logan F.', 'Santa, I think the reindeer are starting a band with the garden gnomes.', 'needs clarification', NULL), (11, 'Carlos P.', 'Thank you, Santa, for making Christmas so special every year!', 'thank you', NULL), (12, 'Zain G.', 'Thank you, Santa, for bringing joy to all the children around the world!', 'thank you', NULL), (13, 'Haruto R.', 'Thank you, Santa, for spreading so much joy and magic every Christmas!', 'thank you', 'approved'), (14, 'Oliver L.', 'Thank you, Santa, for spreading joy and making Christmas extra special!', 'thank you', NULL), (15, 'Luca M.', 'Hi Santa, could I please have the super cool glow-in-the-dark rocket ship?', 'wish list', NULL), (16, 'Samuel C.', 'sorry, Santa, my teddy bear said he wants to be a reindeer this year.', 'needs clarification', NULL); ``` Here's the setup I did to check the data. ```sql SELECT * FROM hotline_messages; ``` ```plaintext $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day3-inserts.sql sqlite> .schema CREATE TABLE hotline_messages ( id INT PRIMARY KEY, caller_name TEXT, transcript TEXT, tag TEXT, status TEXT ); sqlite> .mode table sqlite> SELECT * FROM hotline_messages LIMIT 10; +----+-------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 1 | Saanvi A. | I just found a refrigerator portal that leads to a disco par | possible dragon | spam | | | | ty hosted by dancing llamas—please send help! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 2 | Fatima Q. | Hi Santa, I would love a magical unicorn that lights up at n | wish list | | | | | ight! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 3 | Lillian Z. | Hi Santa, I would love the magical fairy garden set, please! | wish list | approved | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 4 | Carter Y. | Thank you, Santa, for making Christmas so special with your | thank you | approved | | | | wonderful spirit! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 5 | Omar R. | Hi Santa, I would love a rainbow unicorn plushie that has a | wish list | approved | | | | glittery horn! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 6 | Diego Y. | Hi Santa, I would love a magical unicorn plushie that glows | wish list | | | | | in the dark! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 7 | Layla X. | Thank you, Santa, for spreading joy and magic every Christma | thank you | | | | | s! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 8 | Sophia K. | Santa, my cat said she wants to visit the candy cane forest | | | | | | next week. | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 9 | Eli H. | Hi Santa, I would love the magical fairy castle with twinkli | wish list | approved | | | | ng lights! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 10 | Logan F. | Santa, I think the reindeer are starting a band with the gar | needs clarification | | | | | den gnomes. | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ sqlite> ``` So, we just have one table, called `hotline_messages` and it has a few columns like 1. `caller_name` 2. `transcript` 3. `tag` 4. `status` What do we want to do with those? Well! Let's get into the problem statement. ## Problem Here goes the challenge for day 3 > Using the `hotline_messages` table, update any record that has "sorry" (case insensitive) in the transcript and doesn't currently have a status assigned to have a status of "approved". > > Then delete any records where the tag is "penguin prank", "time-loop advisory", "possible dragon", or "nonsense alert" or if the caller's name is "Test Caller". > > After updating and deleting the records as described, write a final query that returns how many messages currently have a status of "approved" and how many still need to be reviewed (i.e., status is NULL). It's divided into 3 parts, so we need three queries? Maybe, I don't want to have a single long query for doing all of these. And after reading it, it seems it should not be a single query, it can be 2 queries, one is for updation and the other for selection after those updates. So, we have to do three things. 1. Find the records which have `sorry` in the transcript text and mark their `status` as `approved` (What a lovely gesture) 2. Find all records with the tags as either `penguin prank`, `time-loop advisory`, `possible dragon`, or `nonsense alert` and even if the `caller_name` is `Test Caller` then delete those records, yes take'em out of my way. 3. After doing those 2 things, we have to count the number of records with `status` as `approved` and the number of records that are still not `approved` (they are in review or the status is `NULL` ) So, let's do them step by step. ### Be generous Let's be generous like Santa says and mark the records with `status` as `approved` whose transcript have the word `sorry` in them. Let those children be gifted their reward of being generous and humble. How do that in SQL, well let's first look at what we are updating. ```sql SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; ``` So, will it be sufficient? I think so. Because * `LIKE` is **case insensitive**, so it can catch `sorry`, `Sorry`, `SoRRY`, `sorrY` * `%` before and after will catch the word `sorry` in middle of the sentence and not necessarily in the start. I can see 104 rows selected with this condition. I always try to check before updation or deletion how many rows are affected. Because, sometimes we start `UPDATE hotline_messages SET status = 'approved'` and forget the where! This gets worse for delete believe me! ```sql SELECT count(*) FROM hotline_messages WHERE transcript LIKE '%sorry%'; SELECT count(*) FROM hotline_messages; ``` ```plaintext sqlite> SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; +------+--------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 16 | Samuel C. | sorry, Santa, my teddy bear said he wants to be a reindeer t | needs clarification | | | | | his year. | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 33 | Jacob F. | sorry, Hi Santa, I would love the magical unicorn plushie th | wish list | | | | | at glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 46 | Jun Y. | sorry, Hi Santa, I would love a magical unicorn stuffed anim | wish list | | | | | al that glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ sqlite> SELECT count(*) FROM hotline_messages WHERE transcript LIKE '%sorry%'; +----------+ | count(*) | +----------+ | 104 | +----------+ sqlite> SELECT count(*) FROM hotline_messages ;--WHERE transcript LIKE '%sorry%'; +----------+ | count(*) | +----------+ | 1067 | +----------+ sqlite> ``` So, once I know `104` rows will be affected out of `1067` I can create the update statement. We want to update the status and set it to `approved` for the rows which we selected just now (have `sorry` in the transcript text) ```sql UPDATE hotline_messages SET status = 'approved' WHERE transcript LIKE '%sorry%'; ``` Now, when we select again ```sql SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; ``` All approved! ```plaintext sqlite> SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; +------+--------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 16 | Samuel C. | sorry, Santa, my teddy bear said he wants to be a reindeer t | needs clarification | approved | | | | his year. | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 33 | Jacob F. | sorry, Hi Santa, I would love the magical unicorn plushie th | wish list | approved | | | | at glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 46 | Jun Y. | sorry, Hi Santa, I would love a magical unicorn stuffed anim | wish list | approved | | | | al that glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ ``` To the next step then ### Remove Spam To reiterate the second part of the challenge > Then delete any records where the tag is "penguin prank", "time-loop advisory", "possible dragon", or "nonsense alert" or if the caller's name is "Test Caller". We basically need to Find all records with the tags as either `penguin prank`, `time-loop advisory`, `possible dragon`, or `nonsense alert` and even if the `caller_name` is `Test Caller` then delete those records, yes take'em out of my way. So, again, select first update or delete later. ```sql SELECT * FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); ``` Here `IN` is a great helper as we can do equivalent of this into a compact statement. ```sql SELECT * FROM hotline_messages WHERE tag = 'penguin prank' OR tag = 'time-loop advisory' OR tag = 'possible dragon' OR tag = 'nonsense alert'; ``` That is a lot of `OR tag =` that is saved by `IN` a list of values. Handy little operator. The count here is `68` ```sql SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); ``` ```plaintext sqlite> SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); +----------+ | count(*) | +----------+ | 68 | +----------+ sqlite> ``` Also we need to check if the `caller_name` is `Test Caller` It could be `OR` here ```sql SELECT * FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` That's it let's count the number of rows we will be deleting soon. ```plaintext SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` So, we have `89` rows to deleted after considering the spamy tags and test callers out. ```plaintext sqlite> SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; +----------+ | count(*) | +----------+ | 89 | +----------+ sqlite> ``` Let's get the spam outta here! ```sql DELETE FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` Phew! Done 89 spammy records removed! Santa might be relieved. ```plaintext sqlite> DELETE FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; sqlite> SELECT changes(); +-----------+ | changes() | +-----------+ | 89 | +-----------+ ``` The changes are done, now we simply have to select and count the things which are approved and in review. ### Count'em down * After doing those 2 things, we have to count the number of records with `status` as `approved` and the number of records that are still not `approved` (they are in review or the status is `NULL` ) So, we need to get the count of 1. Records with `status` as `approved` 2. Records with `status` as `NULL` #### Separate Queries This looks straight forward, you can write two separate queries for doing the things. ```sql SELECT COUNT(*) as approved_count FROM hotline_messages WHERE status = 'approved'; SELECT COUNT(*) as in_review_count FROM hotline_messages WHERE status IS NULL; ``` ```plaintext sqlite> SELECT COUNT(*) as approved_count FROM hotline_messages WHERE status = 'approved'; SELECT COUNT(*) as in_review_count FROM hotline_messages WHERE status IS NULL; +----------------+ | approved_count | +----------------+ | 477 | +----------------+ +-----------------+ | in_review_count | +-----------------+ | 501 | +-----------------+ sqlite> ``` #### Group by Status But however, can we do it in 1 query? Think! There are just 2 types of status right? Let's check ```sql SELECT DISTINCT status FROM hotline_messages; ``` Hmm! 2? `NULL` and `approved` ! ```sql sqlite> SELECT DISTINCT status FROM hotline_messages; +----------+ | status | +----------+ | | | approved | +----------+ ``` So we can simply do the same thing, but just group by `status` right? Like so: ```sql SELECT status, COUNT(*) as count FROM hotline_messages GROUP BY status; ``` And this should give us back the two rows with the count of `NULL` and `approved` . ```plaintext sqlite> SELECT status, COUNT(*) as count FROM hotline_messages GROUP BY status; +----------+-------+ | status | count | +----------+-------+ | | 501 | | approved | 477 | +----------+-------+ ``` Is there a better way? This looks a little wired! Like status is empty (NULL) and it kind of makes a little wired view for people to look at, can we do something different? #### Cases when then else This is simple use case for a `CASE WHEN ... THEN ... ELSE ... END` For each when we can check certain conditions and do certain things or do other thing. In this case, if the status is `approved`, we can increment the count to 1 or we can keep 0, similarly the other when can be used for grouping the count of status being `NULL` . ```sql SELECT COUNT(CASE WHEN status = 'approved' THEN 1 END) AS approved_count, COUNT(CASE WHEN status IS NULL THEN 1 END) AS in_review_count FROM hotline_messages; ``` What this will do is, for each row, we will count up the number of either `approved_count`or `in_review_count` depending on the value of the `status` cell. If that is `approved` we increment the `approved` count else if that is `NULL` we increment the `in_review_count`. Slick! ```sql sqlite> SELECT COUNT(CASE WHEN status = 'approved' THEN 1 END) AS approved_count, COUNT(CASE WHEN status IS NULL THEN 1 END) AS in_review_count FROM hotline_messages; +----------------+-----------------+ | approved_count | in_review_count | +----------------+-----------------+ | 477 | 501 | +----------------+-----------------+ sqlite> ``` That's it from day 3 hopefully Santa is happy, and in sight of getting madder as the elves get dumber.

Advent of SQL 2025 Day 2: Snowballs

Wed, 17 Dec 2025 00:00:00 UTC

## SQLog: Advent of SQL Day 2 Here we are on the day 2 of Advent of SQL As I said in the previous day this is in SQLite so I won't be doing it in the playground. So here is your SQLite playground :) ```sql SELECT 1; ``` From now on no setup straight to the problem! Let's download the .sql file for today's problem to see what data we are playing with. ```sql DROP TABLE IF EXISTS snowball_categories; DROP TABLE IF EXISTS snowball_inventory; CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); ``` Well, neat and clean! Straight pulling it in the sqlite shell. ```plaintext $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day2-inserts.sql sqlite> .schema CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); sqlite> ``` Worked right off the bat. Straight to problem now. If you are working in the playground, do add this full code with inserts in order to get a sense of what the data looks like, note that this below statements is not all data, just sharing to get you a taste of the problem. ```sql DROP TABLE IF EXISTS snowball_categories; DROP TABLE IF EXISTS snowball_inventory; CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); INSERT INTO snowball_categories (id, official_category) VALUES (1, 'frost-flight deluxe'), (2, 'north ridge compact'), (3, 'glacier sphere (xl)'), (4, 'polar precision microball'), (5, 'everfrost training round'), (6, 'arctic blast premium'); INSERT INTO snowball_inventory (id, batch_id, category_name, quantity, status) VALUES (1, 'BATCH-35443-J', 'frost-flight deluxe', 19, NULL), (2, 'BATCH-59767-M', 'frost-flight deluxe', 41, 'incomplete'), (3, 'BATCH-44795-B', 'frost-flight deluxe', 21, 'ready'), (4, 'BATCH-23396-C', 'north ridge compact', 0, 'incomplete'), (5, 'BATCH-88907-A', 'frost-flight deluxe', -2, 'incomplete'), (6, 'BATCH-42662-D', 'frost-flight deluxe', 47, 'needs review'), (7, 'BATCH-37460-V', 'north ridge compact', 43, 'ready'), (8, 'BATCH-21395-S', 'frost-flight deluxe', -2, 'ready'), (9, 'BATCH-36100-E', 'frost-flight deluxe', 46, 'ready'), (10, 'BATCH-64987-H', 'frost-flight deluxe', 43, NULL), (11, 'BATCH-57576-Z', 'melty deluxe', -5, 'ready'), (12, 'BATCH-56025-U', 'snowball v2', 11, 'ready'), (13, 'BATCH-86556-W', 'snowball v2', 12, 'ready'), (14, 'BATCH-83385-N', 'frost-flight deluxe', 38, 'incomplete'), (15, 'BATCH-85156-M', 'prototype x-12', 28, 'incomplete'), (16, 'BATCH-82135-F', 'north ridge compact', 32, 'incomplete'), (17, 'BATCH-10074-T', 'frost-flight deluxe', 49, 'needs review'), (18, 'BATCH-22676-L', 'frost-flight deluxe', 16, 'incomplete'), (19, 'BATCH-31174-R', 'north ridge compact', 33, 'incomplete'), (20, 'BATCH-41385-B', 'frost-flight deluxe', 4, 'ready'), (21, 'BATCH-50404-L', 'frost-flight deluxe', -4, 'needs review'), (22, 'BATCH-92240-F', 'north ridge compact', 20, 'ready'), (23, 'BATCH-29198-J', 'beta test sphere', 0, 'incomplete'), (24, 'BATCH-64987-H', 'glacier sphere (xl)', 18, 'needs review'), (25, 'BATCH-80008-A', 'frost-flight deluxe', 3, 'incomplete'), (26, 'BATCH-88907-A', 'polar precision microball', 48, 'incomplete'), (27, 'BATCH-55830-J', 'north ridge compact', 0, 'needs review'), (28, 'BATCH-69470-A', 'frost-flight deluxe', -3, 'incomplete'), (29, 'BATCH-46211-R', 'frost-flight deluxe', -3, 'ready'), (30, 'BATCH-18675-G', 'glacier sphere (xl)', -1, 'ready'); ``` ## Problem Let's check the challenge of day 2 > Using the `snowball_inventory` and `snowball_categories` tables, write a query that returns valid snowball categories with the count of valid snowballs per category. Your final table should have the columns `official_category` and `total_usable_snowballs`. Sort the output from fewest to most `total_usable_snowballs`. So, we have two tables: 1. snowball categories 2. snowball inventory The snowball categories looks a quite a small table with just the category name and the id which is not really the data, the name is. ```sql SELECT * FROM snowball_categories; ``` Just 6 entries with some names of the categories, the column is official categories. ```sql sqlite> SELECT * FROM snowball_categories; +----+---------------------------+ | id | official_category | +----+---------------------------+ | 1 | frost-flight deluxe | | 2 | north ridge compact | | 3 | glacier sphere (xl) | | 4 | polar precision microball | | 5 | everfrost training round | | 6 | arctic blast premium | +----+---------------------------+ sqlite> ``` And the other table has oddles of data: ```sql SELECT * FROM snowball_inventory limit 10; ``` So, we have quite a few columns, 1. batch\_id 2. category\_name 3. quantity 4. status It has 200,000 rows, that's quite a lot. ```plaintext sqlite> SELECT count(*) FROM snowball_inventory; +----------+ | count(*) | +----------+ | 200000 | +----------+ sqlite> SELECT * FROM snowball_inventory limit 10; +----+---------------+---------------------+----------+--------------+ | id | batch_id | category_name | quantity | status | +----+---------------+---------------------+----------+--------------+ | 1 | BATCH-35443-J | frost-flight deluxe | 19 | | | 2 | BATCH-59767-M | frost-flight deluxe | 41 | incomplete | | 3 | BATCH-44795-B | frost-flight deluxe | 21 | ready | | 4 | BATCH-23396-C | north ridge compact | 0 | incomplete | | 5 | BATCH-88907-A | frost-flight deluxe | -2 | incomplete | | 6 | BATCH-42662-D | frost-flight deluxe | 47 | needs review | | 7 | BATCH-37460-V | north ridge compact | 43 | ready | | 8 | BATCH-21395-S | frost-flight deluxe | -2 | ready | | 9 | BATCH-36100-E | frost-flight deluxe | 46 | ready | | 10 | BATCH-64987-H | frost-flight deluxe | 43 | | +----+---------------+---------------------+----------+--------------+ sqlite> ``` What we have to do? > return valid snowball categories with the count of valid snowballs per category That's a mouthful. The problem is trying to get at counting the number of inventory items per category I think. Because there are quite less categories and tons of inventory records. Also the `category_name` in the `snowball_inventory` table is not trustworthy. > Santa hurried to the snowball storage board, but the situation only got stranger. Whole batches appeared twice. Some batches claimed they had negative snowballs (“a bookkeeping accident,” the elves muttered). Others had a quantity of zero but were still marked “Ready.” And many batches referenced categories that didn’t appear anywhere in the official Snowball Category Guide ## Naive Approach The first approach I see is scan through all the inventory records and check if the category is in the `snowball_category` table and its quantity is positive or more than 0, also I wonder if the status needs to be checked as `ready`. > “We need to know what we actually have left,” Santa said. “Not puddles. Not phantom batches. Real, usable, throw-ready snowballs.” **Real, throw-ready snowballs** Let's check how to do that with simple sub query. ```sql SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready'; ``` OK! What we did? Simply selected all the data from `snowball_inventory` table, in which the category name is matching either of the 6 categories in the `snowball_categories` table. Also the quantity is positive and not zero, also the status is set to be `ready`. This looks naive to me for using subquery because for each 200,000 records we will scan the `snowball_categories` . Ew! Wait its done! We need the count of each category! We need to group by the `category_name` ```sql SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; ``` ```sql sqlite> SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; +-----+---------------+---------------------------+----------+--------+ | id | batch_id | category_name | quantity | status | +-----+---------------+---------------------------+----------+--------+ | 240 | BATCH-55793-L | arctic blast premium | 45 | ready | | 163 | BATCH-75333-O | everfrost training round | 23 | ready | | 3 | BATCH-44795-B | frost-flight deluxe | 21 | ready | | 39 | BATCH-22704-V | glacier sphere (xl) | 37 | ready | | 7 | BATCH-37460-V | north ridge compact | 43 | ready | | 125 | BATCH-81987-E | polar precision microball | 47 | ready | +-----+---------------+---------------------------+----------+--------+ ``` That sort of looks wired right? Why? Because what happens to the quantity? Is that summed? averaged, or what just happened to the batch\_id? No it takes one value out of the `200 000` rows for each category. That's not what we want right? We want this > return valid snowball categories with the count of valid snowballs per category So, we just want the category name and the count of those category. Basically count per category. ```sql SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; ``` Here we are only fetching the `category_name` and the `count` which will bundle up all the counts from the grouped of the category\_name. ```plaintext sqlite> SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | frost-flight deluxe | 952019 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | polar precision microball | 70773 | +---------------------------+------------------------+ sqlite> ``` Not done yet! We need to order by the count. > Sort the output from fewest to most `total_usable_snowballs`. ```sql SELECT category_name, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; ``` We can provide `ORDER BY total_usable_snowballs ASC` but ascending is default. I prefer keeping things default, you can be explicit and mention it as `ASC` to make it clear and readable. ```plaintext sqlite> SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ sqlite> ``` There it is! The solution to the day 2. But Santa, I am using subqueries. Is that fine? ## Joins? We can use joins here, since we just require the count of each category. Which JOIN though? LEFT, RIGHT, INNER? any really? You don't choose the join based on the problem, you define your output columns and then choose the type of join that would give you the result. If the thing that is to be searched from left to right and then its a left join, i.e. join everything from the left. and so on. Here, I need all the `official_category` column right, which is in the `snowball_categories` table. If I assume the `snowball_categories` is on the left, I can join everything for that row in the left to match all the `category_name` rows in the `snowball_inventory` table which would be in the right. Like so. ```sql SELECT snowball_categories.official_category as official_category, SUM(snowball_inventory.quantity) as total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` We JOIN on the condition of `snowball_categories.official_category = snowball_inventory.category_name` which is to say, the `category_name` in the `snowball_inventory` table should match the `official category` from the `snowball_categories` table. Also the other conditions like `quantity` should be more than `0` and the `status` should be `ready`. We will still need to group by and order by as the joins will basically be the filtering criteria in which to select or reject the bad categories. However to group by the category name and obtain the count of each category we need to group by the name of the category. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ ``` This is it! The same thing that we did for the subquery. Just the filtering part is different. How about INNER JOIN? `INNER JOIN` is different from `LEFT JOIN` as it will by default take only rows that match the condition, whereas `LEFT JOIN` will include records/rows from the `LEFT` table even if they are empty or `NULL` . Similar is `RIGHT JOIN`, it will include everything from the RIGHT table even if that is `NULL` so that all the records from the `RIGHT` table are in the result set. For INNER JOIN if we flip the tables end it won't matter, as it only relies on the condition and not on the order of where the tables are placed (left or right). ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` This is with INNER JOIN nothing changed here? did it? except the word `LEFT` is no longer there, the default join is `INNER JOIN` . In our case i don't think we have `NULL` values in the `snowball_categories` table or `snowball_inventory` table with the category name columns. So LEFT AND INNER JOIN won't be different in terms of the results. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ sqlite> ``` However if we decide to make the `snowball_inventory` table as the `Left` table then does it make a difference? Well we don't want that to be the left table if we are doing a `LEFT` JOIN as it might list all the wrong categories in the result list as well even if they are not in the `official_category` those will be 0 or NULL but that would make a mess. We instead can do a RIGHT JOIN. With the tables order as 1. snowball\_inventory 2. snowball\_categories And we will RIGHT JOIN that is select all the columns from the right to be in the result set even if the condition is not true. It will be NULL. But the case is not true for here, as there are only 6 non-NULL category names in the snowball\_categories table. ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ ``` However if we try to use `LEFT JOIN` with `snowball_inventory` as the `LEFT` table, we might get something wired. ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory LEFT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` We have 6 and 1 more category as empty ? That category is for all the rows that didn't met the criteria of JOIN but still as asked to `LEFT JOIN` it will try to show the results for all the rows in the left table which is the `snowball_inventory` table. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory LEFT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | | | 1692814 | +---------------------------+------------------------+ ``` ## Timer Phew! Let's see which one of them is the best one Let's see the timings: ```plaintext .timer on ``` ### SUBQUERY ```plaintext sqlite> SELECT category_name, SUM(quantity)AS total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | category_name | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.077 user 0.073886 sys 0.002942 sqlite> ``` ### LEFT JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.182 user 0.171433 sys 0.009907 sqlite> ``` ### INNER JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.072 user 0.069623 sys 0.001929 sqlite> ``` ### RIGHT JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.151 user 0.149690 sys 0.001957 sqlite> ``` Time wise ranking: 1. INNER JOIN 2. SUBQUERY 3. RIGHT JOIN 4. LEFT JOIN That's quite quick, not much can be measured here. I am sure there are other ways to do this. But stopping here for today. Explored JOINs a little. See you for the day 3

Advent of SQL 2025: Wish List

Tue, 16 Dec 2025 00:00:00 UTC

## Learning SQLite: Advent of SQL Day 1 I am trying to learn SQLite, I want to understand that database. It's quite simple yet the whole world uses it for various kinds of things ranging from developers' toy database to spaceships. What a tiny engineering marvel! I am happy to see this happening: [Advent of SQL](https://databaseschool.com/series/advent-of-sql-videos-308) What a better time to learn more. I guess I want to start by exploring all the specificities of the INSERT statement in SQLite after exploring most of the things of the CREATE TABLE statement. But here I am jumping to this. Why? Because I want to solve something first before exploring other branches. Today I am going to try to solve the day 1 part. > **NOTE:** I would be using my local sqlite db for this or a playground on this for testing. I am not going to use the browser-based playground attached in the databaseschool.com app for a reason. I want to use SQLite. The database is some form of Postgres; I don't mind using it, but I want to do it in SQLite. I have a playground on my blog for SQLite, you can try it out here: ```sql SELECT 1; ``` It uses an embedded SQLite version (3.49.1) with sql-js as a wasm extension: [sql.js v1.13.0](https://github.com/sql-js/sql.js/releases/tag/v1.13.0) Back to the problem elves! ## Setup This is the first day, so advent calendar usually requires some setup or preparation for the rest of the days. Luckily it's optional for you if you are doing it in the playground of database school or in PostgreSQL Database. We have some .sql files as input for creation and insertions of tables and rows in the database. It's for constructing the schema (tables) and populating the rows that the problem requires us to do. The SQL looked something simpler like: ```sql DROP TABLE IF EXISTS wish_list CASCADE; CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (1, 'James A.', ' BLUEY SUPERMARKET PLAY SET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (2, 'Sade C.', 'lego star wars set '); ``` There are around 499,000 rows! However when I tried to read directly into a SQLite shell as: ``` .read day1-wish-list.sql ``` It got an error for parsing the DROP TABLE statement: ``` sqlite> .read day1-wish-list.sql Parse error near line 11: near "CASCADE": syntax error DROP TABLE IF EXISTS wish_list CASCADE; error here ---^ ``` Obviously it was designed for Postgres. It won't work in SQLite. SQLite is minimal. It might not have everything that PostgreSQL has, but PostgreSQL might have everything that SQLite has (maybe but not as is). So, we need to remove the CASCADE, which is an option to decide what to do with the related data rows when a relation is removed. In this case, it is cascading—deleting all the other related data points or records in the related tables. SQLite doesn't have options to modify the relations for the DROP TABLE statement. It has it for CREATE TABLE with the foreign key constraint. Now we need to remove it. It can't be in the DROP TABLE statement for SQLite database: ```sql DROP TABLE IF EXISTS wish_list; ``` Now, let's check by running the queries again: ``` .read day1-wish-list.sql ``` That works! ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day1-wish-list_sqlite1.sql sqlite> .schema CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); ``` But this looks weird: ```sql CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); ``` BIGSERIAL is not a datatype in SQLite. It might be in PostgreSQL. But does it matter? In SQLite, if the table is not STRICT, it doesn't matter what type the column is or even if it is NOT specified of any type. That's fine. INSERT some rows shall we? ```sql INSERT INTO wish_list (id, child_name, raw_wish) VALUES (1, 'James A.', ' BLUEY SUPERMARKET PLAY SET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (2, 'Sade C.', 'lego star wars set '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (3, 'Juan Q.', ' SCOOTER '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (4, 'Samir S.', ' LEGO STAR WARS SET '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (5, 'Priya E.', 'shaved ice machine '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (6, 'Henry L.', ' mini brands fill the fridge'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (7, 'Ayumi C.', 'VR HEADSET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (8, 'Juan Y.', 'BARBIE DREAMHOUSE '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (9, 'Priya O.', ' VR HEADSET '); ``` ```sql SELECT * FROM wish_list; ``` But I want to know what is the type of `wish_list.id`? ```sql SELECT distinct(typeof(id)) FROM wish_list; ``` Here's the output: ``` sqlite> SELECT distinct(typeof(id)) FROM wish_list; integer ``` It is integer, because of type affinity, I guess. Good work SQLite. Looks like the data is fine. Now to the problem. ## Problem [Link to the challenge](https://databaseschool.com/series/advent-of-sql-videos-309-text-challenge-using-the-and-get-started) **Challenge:** Using the wish_list table, count how many times each cleaned toy name appears, from most requested to least requested. Return the results in two columns: wish and count. Make sure the wish results have no extra leading or trailing spaces and are all lowercase. So, simply we need two columns: - **wish** (the text) - **count** (the number of times that wish is wished) ### Subtleties > Some children had typed extra spaces. Some wrote in ALL CAPS. Some had letters that danced between cases like playful snowflakes. > > I know there are some issues with spelling, the extra spaces, or the funny capitalization, but I just need to know what the children truly meant. So, we need to either use lower caps or upper caps the wish text and trim off the space. SCALAR FUNCTIONS!! ### Scalar Functions [SQLite Core Functions](https://sqlite.org/lang_corefunc.html) I read through the list of around ~70 of them, most of them are kind of the same with different parameters. The one that I found relevant are: - [LOWER](https://sqlite.org/lang_corefunc.html#lower) - [TRIM](https://sqlite.org/lang_corefunc.html#trim) That's it, right? Convert into LOWER (or UPPER) and TRIM off the spaces. ```sql SELECT LOWER(TRIM(raw_wish)) FROM wish_list; ``` Don't run just yet! ```sql SELECT LOWER(TRIM(raw_wish)) FROM wish_list LIMIT 100; ``` Looks good. Now to the next step. ### Grouping and Counting We need to count them i.e. to group by the wish. > GROUP BY: What group by does is that it condenses the rows of certain column into a singular column for instance if there are 10 entries for "lego star wars set" adding a group by wish will create a singular entry for that wish and we can then perform operations like sum, count, average and all of that on other rows ```sql SELECT LOWER(TRIM(raw_wish)) AS wish, count(*) AS count FROM wish_list GROUP BY wish; ``` Here we are grouping by wish because we don't want 10 entries of "lego star wars set" we just want one common entry to view the unique wishes. Also by using `COUNT(*)` we are counting the instances of each row. As I said, the multiple rows with the same wish are squished into a single row. So now we can use aggregate functions like count, sum, in our case we want to count how many instances of those particular wish are. ### Ordering Results Does that solve it? Mostly, just need the ORDER BY now. Because we also need to order the results: ```sql SELECT LOWER(TRIM(raw_wish)) AS wish, count(*) AS count FROM wish_list GROUP BY wish ORDER BY count DESC; ``` Perfect? Probably. > What ORDER BY does is that it just determines which way the row should be aligned based on what column and how i.e. the column name and either ASC(ending) or DESC(ending). Here we have ordered by count so that we can filter the most wished toy or least wished toy at the top and increasing or decreasing order of it. ### Results Now with the mode table: ``` sqlite> .mode table sqlite> SELECT LOWER(TRIM(raw_wish)) as wish, count(*) as count FROM wish_list group by wish order by count desc; +-----------------------------+-------+ | wish | count | +-----------------------------+-------+ | lego city f1 car | 32893 | | barbie dreamhouse | 32785 | | nerf blaster | 32746 | | lego star wars set | 32611 | | beyblade battle arena | 29564 | | magna-tiles pet playhouse | 29529 | | bluey supermarket play set | 26292 | | lego friends amusement park | 25982 | | pokemon trainer box | 25968 | | duplo building set | 23005 | | mini brands fill the fridge | 22965 | | electric toy train set | 22885 | | toniebox audio player | 19529 | | scooter | 19496 | | vr headset | 16468 | | squishmallows | 16304 | | shaved ice machine | 16263 | | drone for kids | 13151 | | coding robot | 13025 | | headphones | 13006 | | interactive robot dog | 9770 | | fidget spinner | 3590 | | yo-yo | 3565 | | slime kit | 3553 | | littlest pet shop playset | 3543 | | chatter telephone | 3527 | | fingerlings robot monkey | 3511 | | rubik's revolution | 3474 | +-----------------------------+-------+ sqlite> ``` --- Day 1 done, moving on to day two by helping those pesky elves tomorrow. I am amazed at how stupider problems humans create with those elves as target. Just kidding, humans are elves :) Happy Coding :) Happy Squealing

SQLITE SQL: Create Table with STRICT Option

Fri, 12 Dec 2025 00:00:00 UTC

We have seen how to create `TABLE` loosely Not adhering to the types. Because if we create a table with column name of type text, and insert a integer, it will happily store it as text. It is very flexible as we saw. So, in such `CREATE TABLE` statement in SQLite, without the `STRICT` constraint, the types don't matter. If you created a table with column type as XYZ it will accept it, because it really doesn't see that. It will see the data coming in from the insert statement and store it whatever it thinks is the best one for that piece of data. Look at the below example: ```sql CREATE TABLE t1 (n xyz); INSERT INTO t1 values(4); INSERT INTO t1 values("gg"); SELECT rowid, n, typeof(n) FROM t1; ``` ``` rowid n typeof(n) 1 4 integer 2 gg text ``` See? The column type, it doesn't matter. Unless it's strict or any constraints, or generated conditions have been added. The STRICT table option Let's quote from the documentation what it means > * Every column definition must specify a datatype for that column. > > * The freedom to specify a column without a datatype is removed. > > * The datatype must be one of the following: > > * INT > > * INTEGER > > * REAL > > * TEXT > > * BLOB > > * ANY > > * The [PRAGMA integrity\_check](https://sqlite.org/pragma.html#pragma_integrity_check) and [PRAGMA quick\_check](https://sqlite.org/pragma.html#pragma_quick_check) commands check the type of the content of all columns in STRICT tables and show errors if anything is amiss. > There are other nuances of the STRICT table options and the kind of constraint that you put on the columns, but that requires studying very specific examples. We'll check those nuances later. For now though, we need to understand how to create a strictly typed table, and what the strict option adds to the table. ```sql CREATE TABLE users( name TEXT, age INT, credits REAL, profile_pic BLOB ) STRICT; ``` So, we have all the actual possible types we can use in a table column when defining a table. If you don't provide an column type, or provide any other type than `TEXT`, `INT` or `INTEGER`, `REAL`, `BLOB`, or `ANY` (don't put any, you lose the purpose of strict) it won't compile and execute the table creation. You need to provide a valid type among the 5 types. However if you try to create a strict table with wrong column type or no column type. ```sql CREATE TABLE t1 (t) STRICT; -- Error: missing datatype for t1.t CREATE TABLE t1 (t something) STRICT; -- Error: unknown datatype for t1.t: "something" ``` Without STRICT it works as usual: ```sql CREATE TABLE t1 (t something); INSERT INTO t1 values(123), ('abc'), (X''), (123.45); SELECT t, typeof(t) FROM t1; ``` ``` t typeof(t) 123 integer abc text blob 123.45 real ``` Now back to the original example: Insert a couple of rows: ```sql -- All are NULL Values INSERT INTO users DEFAULT VALUES; INSERT INTO users (name, age, credits, profile_pic) VALUES ( 'Alice', 30, 100.0, X'89504E470D0A1A0A' ); ``` This will insert two rows, the first one, all the columns will be `NULL`. If you look at the type of these statement. Those will be as per the table schema, consistent for all rows. ``` name typeof(name) age typeof(age) credits typeof(credits) profile_pic typeof(profile_pic) null null null null Alice text 30 integer 100 real 137,80,78,71,13,10,26,10 blob ``` This has rightly added NULL type for the null values but when the data is in the row, it forces that type stated in the schema of the table. Now, if we try to mess up the column data, it won't work ```sql INSERT INTO users (name, age, credits, profile_pic) VALUES (34, '4', 8, 123); -- Error: cannot store INT value in BLOB column users.profile_pic INSERT INTO users (name, age, credits, profile_pic) VALUES (34, '4', 8, ''); -- Error: cannot store TEXT value in BLOB column users.profile_pic INSERT INTO users (name, age, credits, profile_pic) VALUES (34, 'abc', 8, X''); -- Error: cannot store TEXT value in INT column users.age ``` This will work, as type affinity and the conversion is possible within the column types here. ``` INSERT INTO users (name, age, credits, profile_pic) VALUES (CAST(34 AS INT), '3', 8, X''); ``` But if some data is not able to convert into that strict type, it will fail the constraint of strict column type. For instance - `123` or `""` is not force convertible to BLOB which is binary large object. We need to parse it with X'' strings for some raw data to make it a BLOB like object in SQLite. - `abc` is not convertible/casteble to INTEGER or REAL Value. So, the strict type is actually strict as we see the pattern repeating in SQLite. > It is flexible till you allow it to be, you can at anytime change the lever and make it strict This is true for column-row level type checking with the STRICT table option while creating table.

SQLite SQL: Create Table with Select

Sun, 28 Sep 2025 00:00:00 UTC

## Creating table with Select Query Sometimes you want to create a new table based on data that already exists,maybe you’re archiving old rows, generating a summary, cloning a table, or transforming data into a new structure. Doing this might require you to write a `CREATE TABLE` statement with all column definitions, then running one or more `INSERT INTO` statements to populate it. That’s a lot of work. Well, SQL is more flexible then you might think, Instead of manually defining columns and inserting data, you write one statement that both builds the table and fills it with the rows returned by your `SELECT` query. This makes it incredibly useful for backups, and whatever you are doing (hopefully not taking down the prod db). We have CTAS, `CREATE TABLE AS SELECT` statements. This basically means to take whatever the SELECT gives, and turn it into a table. ## Creating CTAS Lets's start with a simple example, we will use our goodol users table first and populate it with some data. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER NOT NULL ); ``` This would create a table called `users` with the columns `id`, `name`, and `age`. The `id` column is the primary key, and the `name` and `age` columns are not null. Let's insert some data into the table. ```sql INSERT INTO users (name, age) VALUES ('abc', 20), ('def', 30), ('ghi', 40); ``` This would insert three rows into the `users` table, with the names `abc`, `def`, and `ghi`, and the ages 20, 30, and 40. Now, let's use the `CREATE TABLE AS SELECT` statement to create a new table called `users_copy` that is a copy of the `users` table. ```sql CREATE TABLE users_copy AS SELECT * FROM users; ``` This would create a new table called `users_copy` that is a copy of the `users` table, with the same columns and data. It will copy the same column structure and data from the `users` table to the `users_copy` table. ```sql SELECT * FROM users_copy; ``` Let's check the schema of the new table `users_copy` ```sql PRAGMA table_info(users_copy); PRAGMA table_info(users); ``` This would print the schema of the `users_copy` table. ``` PRAGMA table_info(users_copy); +-----+------+------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------+------+---------+------------+----+ | 0 | id | INT | 0 | | 0 | | 1 | name | TEXT | 0 | | 0 | | 2 | age | INT | 0 | | 0 | +-----+------+------+---------+------------+----+ PRAGMA table_info(users); +-----+------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------+---------+---------+------------+----+ | 0 | id | INTEGER | 0 | | 1 | | 1 | name | TEXT | 1 | | 0 | | 2 | age | INTEGER | 1 | | 0 | +-----+------+---------+---------+------------+----+ ``` You can see the difference here, the `users.id` column is a primary key in the `users` table, however in the `users_copy` table it isn't. Also the constraints like `notnull` are not reciprocated in the `users_copy` table. So, is this a good way to create a copy of a table? Maybe, its just copying the data and the bare-bone structure of the table and not the schema. ```sqlite sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER NOT NULL ); sqlite> INSERT INTO users (name, age) VALUES ('abc', 20), ('def', 30), ('ghi', 40); sqlite> .mode table sqlite> SELECT * FROM users; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | abc | 20 | | 2 | def | 30 | | 3 | ghi | 40 | +----+------+-----+ sqlite> CREATE TABLE users_copy AS SELECT * FROM users; sqlite> SELECT * FROM users_copy; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | abc | 20 | | 2 | def | 30 | | 3 | ghi | 40 | +----+------+-----+ sqlite> PRAGMA table_info(users_copy); +-----+------+------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------+------+---------+------------+----+ | 0 | id | INT | 0 | | 0 | | 1 | name | TEXT | 0 | | 0 | | 2 | age | INT | 0 | | 0 | +-----+------+------+---------+------------+----+ sqlite> PRAGMA table_info(users); +-----+------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------+---------+---------+------------+----+ | 0 | id | INTEGER | 0 | | 1 | | 1 | name | TEXT | 1 | | 0 | | 2 | age | INTEGER | 1 | | 0 | +-----+------+---------+---------+------------+----+ sqlite> ``` ## What CTAS do? It will create a new table called `users_copy` that is a copy of the `users` table, with the same columns and data. However, it just copies the data and structure but not the entire schema. What I means is that if you have constraints and indexes, the select statement won't copy it over to the new table. So, it will not copy the following; - Constraints - indexes - Triggers However, it will copy everything in the select statement's output. So, you can add custom columns, computed columns, etc. You can also filter the select statement with `WHERE` and `ORDER BY` statements in order to store the limited set of data that you wished to store. ## Copying only the structure If you only wanted to only copy the structure and not the entire data, you can invalidate any rows from the select statement with the `WHERE` clause. This will just create columns in the new table. Giving you a fresh copy of the new table, without any data. ```sql CREATE TABLE users_copy AS SELECT * FROM users WHERE 0; ``` This will create a new table called `users_copy` that is a copy of the `users` table, but without any data. ```sqlite sqlite> CREATE TABLE users_copy AS SELECT * FROM users WHERE 0; Parse error: table users_copy already exists CREATE TABLE users_copy AS SELECT * FROM users WHERE 0; ^--- error here sqlite> CREATE TABLE IF NOT EXISTS users_copy AS SELECT * FROM users WHERE 0; sqlite> SELECT * FROM users_copy; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | abc | 20 | | 2 | def | 30 | | 3 | ghi | 40 | +----+------+-----+ sqlite> DROP TABLE users_copy; sqlite> CREATE TABLE IF NOT EXISTS users_copy AS SELECT * FROM users WHERE 0; sqlite> SELECT * FROM users_copy; sqlite> PRAGMA table_info(users_copy); +-----+------+------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+------+------+---------+------------+----+ | 0 | id | INT | 0 | | 0 | | 1 | name | TEXT | 0 | | 0 | | 2 | age | INT | 0 | | 0 | +-----+------+------+---------+------------+----+ sqlite> ``` As you can see, we need to actually drop the table in order to create a new table form scratch just like any normal table in sqlite. So, there are a few more gotchas and details of this, than I thought to be, will be taking a closer look at each one by one in the next few posts.

SQLite SQL: Create Table If Not Exists

Fri, 26 Sep 2025 00:00:00 UTC

## Creating table if not already exists The `CREATE TABLE` has one clause that we can add to create table if it doesn't exist already. So this comes handy if you already have created a table and instead of throwing an error it simply gracefully handles the query and doesn't re-create the table. Let's first creat a table, as usual, we'll use the goodol `users` table. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); ``` Let's insert some data into the table. ```sql INSERT INTO users (name) VALUES ('abc'), ('def'); ``` Ok, the table `users` exist and also has some data in it. ```sql SELECT * FROM users; ``` Now, if we try to create the same table again, it will throw an error. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL, password TEXT NOT NULL ); ``` It will result in `users` table already exists error. You cannot re-create a table that already exist, you need to drop the table and create again, or alter any columns you wanted to if that is possible. Or, if you just want to create a table only if it doesn't already exists, then you can add the `IF NOT EXISTS` clause: ```sql CREATE TABLE IF NOT EXISTS users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL, password TEXT NOT NULL ); ``` Now, if we try to create the table again, it will not throw any error. However, it won't re-create the `users` table, it will just skip the creation. The table would be as is. ```sqlite sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); sqlite> INSERT INTO users (name) VALUES ('abc'), ('def'); sqlite> .mode table sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | +----+------+ sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL, password TEXT NOT NULL ); Parse error: table users already exists CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, e ^--- error here sqlite> CREATE TABLE IF NOT EXISTS users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, email TEXT NOT NULL, password TEXT NOT NULL ); sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | +----+------+ sqlite> .schema users CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); sqlite> ``` ## Where can we use this? - Accidental cases: Avoiding creating a table that already exists. - Code Generation tools: If you are using schema-based code generation tools like [sqlc](https://github.com/sqlc-dev/sqlc), you might keep on re-running the generation code again and again, to avoid the error statements, you can use this to stop table from re-creating or erroing queries.

SQLite SQL: Create Temporary Table

Wed, 24 Sep 2025 00:00:00 UTC

## Temporary Table in SQLite A Temporary table as the name suggests, is a temporary table. It only resisdes on the database until the current session of the database. In case of sqlite, the temporary table is available in the session until the file reader or session driver closes the connection. The table is created in a separate database file called `temp` that is stored in your temporary path. ## Create a temporary table To create the temporary table, simply use the `TEMP` or `TEMPORARY` keyword before the `TABLE` in `CREATE TABLE` expression. So, `CREATE TEMP TABLE` or `CREATE TEMPORARY TABLE` will be the notation to create a temporary table in sqlite. ```sql CREATE TEMP TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); ``` This will create a temporary table called `users` in the `temp` database. You can insert, update, delete, query and relate to other tables just like a normal table. ```sql INSERT INTO users (name) VALUES ('abc'), ('def'), ('ghi'); ``` ```sql SELECT * FROM users; ``` ```sql UPDATE users SET name = 'xyz' WHERE name = 'abc'; SELECT * FROM users; ``` ```sql DELETE FROM users WHERE name = 'def'; ``` ```sql CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT, content TEXT, user_id INTEGER, FOREIGN KEY (user_id) REFERENCES users (id) ); ``` ```sqlite $ sqlite3 myusers.db sqlite> CREATE TEMP TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); sqlite> .tables temp.users sqlite> INSERT INTO users (name) VALUES ('abc'), ('def'), ('ghi'); sqlite> .mode table sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | | 3 | ghi | +----+------+ sqlite> UPDATE users SET name = 'xyz' WHERE name = 'abc'; SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | xyz | | 2 | def | | 3 | ghi | +----+------+ sqlite> DELETE FROM users WHERE name = 'def'; sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | xyz | | 3 | ghi | +----+------+ sqlite> CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT, content TEXT, user_id INTEGER, FOREIGN KEY (user_id) REFERENCES users (id) ); sqlite> .tables posts temp.users sqlite> INSERT INTO posts(title, content, user_id) VALUES('banger post', 'ai slop', 1); sqlite> INSERT INTO posts(title, content, user_id) VALUES('ai slop', 'banger content', 3); sqlite> SELECT * FROM posts; +----+-------------+----------------+---------+ | id | title | content | user_id | +----+-------------+----------------+---------+ | 1 | banger post | ai slop | 1 | | 2 | ai slop | banger content | 3 | +----+-------------+----------------+---------+ sqlite> ``` So, this creates a temporary table, but what if we already have a `users` table in the database? > You can notice, when I query all the tables with the dot-command `.tables`, it adds a prefix of `temp.` to the temporary table. That is an indication of the table being stored in a temporary database file. Hence if there were two `users` table created, we would be able to identify it with the `temp` and the `main` as the database name. The `main` is optional, but if you want to refer to the `temporary` table then explicitly mention it as `temp.users` or `temp.` However, if you try to perform anything on the actual table, you will have to explicitly mention the `main` keyword, as the temp table will take precedence over the table in the main database. I reponed the database as `myusers.db` , now the `users` table which was the temporary table, is gone. ```sql .tables ``` This only shows, `posts` table, but the temporary table was destroyed. Let's create back the `users` table as a temporary table in the database. ```sql CREATE TEMP TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); ``` This will create the table in the `temp` database. Now, if we check the tables, we will see the `temp.users` table. ```sql .tables ``` Let's create a new table `users` in the main database. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); ``` This would create a `users` table that will be stored permanently in the main database i.e. the `myusers.db` in my case to the file. Let's view what the `.tables` output, it shows 3 tables now - `posts` - `temp.users` - `users` Now, here, if you insert bunch of different values into the `users` table. ```sql INSERT INTO users (name) VALUES ('abc'), ('def'), ('ghi'); ``` And query the `users` table ```sql SELECT * FROM users; ``` Which table is that inserted and queried? The `temp.users` table, because the precedence of temporary table is higher than the main table. So, let's try to query the `temp.users` table. ```sql SELECT * FROM temp.users; ``` As you can see the table has the records. But if we try to query the `main.users` table, it will have no records. ```sql SELECT * FROM main.users; ``` This is empty. As, expected, just the `users` table will refer to th temporary table and not the main table. So, let's insert different values in the `main.users` table. ```sql INSERT INTO main.users (name) VALUES ('pqr'), ('stu'); ``` And query the `main.users` table. ```sql SELECT * FROM main.users; ``` This will show the inserted records in the `main.users` table. Here's the SQLog :) ```sqlite $ sqlite3 myusers.db sqlite> .tables posts sqlite> CREATE TEMP TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); sqlite> .tables posts temp.users sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); sqlite> .tables posts temp.users users sqlite> .schema temp.users CREATE TABLE temp.users ( id INTEGER PRIMARY KEY, name TEXT ); sqlite> .schema main.users CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); sqlite> SELECT * FROM users; sqlite> SELECT * FROM temp.users; sqlite> INSERT INTO users (name) VALUES ('abc'), ('def'), ('ghi'); sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | | 3 | ghi | +----+------+ sqlite> SELECT * FROM temp.users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | | 3 | ghi | +----+------+ sqlite> SELECT * FROM main.users; sqlite> INSERT INTO main.users (name) VALUES ('pqr'), ('stu'); sqlite> SELECT * FROM main.users; +----+------+ | id | name | +----+------+ | 1 | pqr | | 2 | stu | +----+------+ sqlite> SELECT * FROM temp.users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | | 3 | ghi | +----+------+ sqlite> SELECT * FROM users; +----+------+ | id | name | +----+------+ | 1 | abc | | 2 | def | | 3 | ghi | +----+------+ sqlite> ``` So, to sum up: - `CREATE TEMP TABLE` or `CREATE TEMPORARY TABLE` will create a temporary table in sqlite. - The temporary table is available in the session until the file reader or session driver closes the connection. - The temporary table is stored on the separate temporary file - The temporary table is prefixed with the `temp` table. - If there are two tables, one temporary and one permanent, with the same name, then the temporary table will be prefered unless `main` is prefixed to the table name.

SQLite SQL: Collate Column Modifier

Mon, 22 Sep 2025 00:00:00 UTC

## Collate Column Modifier Collate is a modifier in SQL that specifies the column how to arrange and compare the values that would be populated in the rows. So, for a given table, we can specify a few modifiers that would let SQL decide how to handle the values. ## Adding Collate Modifier To add a collate modifier, we can use the `COLLATE` keyword, followed by the collation name. There are 3 collations available in SQLite: - BINARY - NOCASE - RTRIM These basically define how the column value might be treated in later comparisons or populating in querying. Let's quickly understand each with the basic example ## BINARY The `BINARY` modifier is the **default collation** in SQLite. It compares values **byte by byte**, meaning it is case-sensitive and space-sensitive. Let's create a demo table: ```sql CREATE TABLE users ( name TEXT COLLATE BINARY ); ``` Now, let's insert some values ```sql INSERT INTO users (name) VALUES ('jane'), ('John'), ('JANE'), ('Jane '); ``` Here, I have inserted 4 names in the `users` table. I have deliberately inserted a mix of cases. Let's now see how the values are populated in the table. ```sql SELECT * FROM users; ``` We see that there are `4` rows rightly being populated. ```sql SELECT * FROM users WHERE name = 'jane'; ``` Now, we see that only the `jane` name is being populated. This is because the `BINARY` modifier is not case sensitive and space sensitive. ```sqlite sqlite> CREATE TABLE users ( name TEXT COLLATE BINARY ); sqlite> INSERT INTO users (name) VALUES ('jane'), ('John'), ('JANE'), ('Jane '); sqlite> SELECT * FROM users; +---------+ | name | +---------+ | jane | | John | | JANE | | Jane | +---------+ sqlite> SELECT * FROM users WHERE name = 'jane'; +------+ | name | +------+ | jane | +------+ ``` This is the modifier added by default to the columns that have `TEXT` affinity (not necessarily the `TEXT` type). ## NOCASE The `NOCASE` modifier is used to ignore the case of the values in the column. When comparing the two values of the same column, if that values are comparable as strings, then the case is not considered. Let's create a typical `users` table with a `name` column ```sql CREATE TABLE users ( name TEXT COLLATE NOCASE ); ``` Here, we have added a `COLLATE NOCASE` modifier to the `name` column. Let's insert some values: ```sql INSERT INTO users (name) VALUES ('John'), ('jane'), ('JANE'); ``` Here, I have inserted 3 names in the `users` table. I have deliberately inserted a mix of cases. Let's now see how the values are populated in the table. The `Jane` name is in lowercase as well as uppercase. ```sql SELECT * FROM users; ``` We see that there are `3` rows rightly being populated. Now, what would happen if the user queries for `jane`? ```sql SELECT * FROM users WHERE name = 'jane'; ``` This comes with both the `jane`s the `jane` with lowercase as well `JANE` with uppercase. This is because the `NOCASE` modifier is not case sensitive. ```sqlite sqlite> CREATE TABLE users ( name TEXT COLLATE NOCASE ); sqlite> INSERT INTO users (name) VALUES ('John'), ('jane'), ('JANE'); sqlite> .mode table sqlite> SELECT * FROM users; +------+ | name | +------+ | John | | jane | | JANE | +------+ sqlite> SELECT * FROM users WHERE name = 'jane'; +------+ | name | +------+ | jane | | JANE | +------+ sqlite> ``` That is the basic idea of how the `NOCASE` collation works. ## RTRIM The `RTRIM` modifier is used to remove the trailing spaces from the values in the column. Let's create a typical `users` table with a `name` column ```sql CREATE TABLE users ( name TEXT COLLATE RTRIM ); ``` Here, we have added a `COLLATE RTRIM` modifier to the `name` column. Let's insert some values: ```sql INSERT INTO users (name) VALUES ('John'), ('jane'), ('JANE'), ('Jane '), ('jane '); ``` Here, I have inserted 5 names in the `users` table. I have deliberately inserted a mix of cases. Let's now see how the values are populated in the table. ```sql SELECT * FROM users; ``` We see that there are `5` rows rightly being populated. ```sql SELECT * FROM users WHERE name = 'jane'; ``` We see 2 rows being populated. This is because the `RTRIM` modifier is removing the trailing spaces. The first row is the `jane` name without any spaces, the 2nd row that we added. The second row is the `jane` name with the trailing space, the last one that was added. ```sql sqlite> CREATE TABLE users ( name TEXT COLLATE RTRIM ); sqlite> INSERT INTO users (name) VALUES ('John'), ('jane'), ('JANE'), ('Jane '), ('jane '); sqlite> SELECT * FROM users; +---------+ | name | +---------+ | John | | jane | | JANE | | Jane | | jane | +---------+ sqlite> SELECT * FROM users WHERE name = 'jane'; +---------+ | name | +---------+ | jane | | jane | +---------+ sqlite> SELECT rowid, * FROM users WHERE name = 'jane'; +-------+---------+ | rowid | name | +-------+---------+ | 2 | jane | | 5 | jane | +-------+---------+ sqlite> ``` That is the basic idea of how the `RTRIM` collation works.

SQLite SQL: Many to Many Table without RowID

Sat, 20 Sep 2025 00:00:00 UTC

## Many to Many Relation We saw the basic example of [Many-To-Many](https://www.meetgor.com/sqlog/sqlite-many-to-many-relations/) Relation in the second last post from this, there we just focused on the concept of the relation and not so much on the structure of the junction table. Let's take a look at the schema again: ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL ); CREATE TABLE author_post ( user_id INTEGER NOT NULL, post_id INTEGER NOT NULL, PRIMARY KEY (user_id, post_id), FOREIGN KEY (user_id) REFERENCES users (id), FOREIGN KEY (post_id) REFERENCES posts (id) ); ``` Let's now populate the tables. ```sql -- adding authors/users INSERT INTO users(name) VALUES ('Glauber'), ('Jamie'), ('Pekka'); -- adding posts written by pekka INSERT INTO posts(title, content) VALUES ('Switching to Zig from Rust', 'I love C'); INSERT INTO posts(title, content) VALUES ('RAG in SQLite', 'AI first database'); INSERT INTO author_post(user_id, post_id) VALUES (3, 1), (3, 2); -- adding posts written by glauber INSERT INTO posts(title, content) VALUES ('Rewriting SQLite', 'We are no more a sqlite-fork'); INSERT INTO posts(title, content) VALUES ('Offline Writes in SQLite', 'Lets sync'); INSERT INTO author_post(user_id, post_id) VALUES (1, 3), (1, 4); -- adding a post co-authored by pekka and glauber INSERT INTO posts(title, content) VALUES('Limbo', 'SQLite in Rust'); INSERT INTO author_post(user_id, post_id) VALUES (3, 5); INSERT INTO author_post(user_id, post_id) VALUES (1, 5); ``` This has now populated the following tables: ```sql SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id; ``` This has 5 posts, two written by `Pekka`, two written by `Glauber`, and one co-authored by both `Pekka` and `Glauber`. Here's the SQLog: ``` sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL ); CREATE TABLE author_post ( user_id INTEGER NOT NULL, post_id INTEGER NOT NULL, PRIMARY KEY (user_id, post_id), FOREIGN KEY (user_id) REFERENCES users (id), FOREIGN KEY (post_id) REFERENCES posts (id) ); sqlite> .mode table sqlite> INSERT INTO users(name) VALUES ('Glauber'), ('Jamie'), ('Pekka'); sqlite> INSERT INTO posts(title, content) VALUES ('Switching to Zig from Rust', 'I love C'); INSERT INTO posts(title, content) VALUES ('RAG in SQLite', 'AI first database'); SELECT * FROM posts; +----+----------------------------+-------------------+ | id | title | content | +----+----------------------------+-------------------+ | 1 | Switching to Zig from Rust | I love C | | 2 | RAG in SQLite | AI first database | +----+----------------------------+-------------------+ sqlite> INSERT INTO posts(title, content) VALUES ('Rewriting SQLite', 'We are no more a sqlite-fork'); INSERT INTO posts(title, content) VALUES ('Offline Writes in SQLite', 'Lets sync'); SELECT * FROM posts; +----+----------------------------+------------------------------+ | id | title | content | +----+----------------------------+------------------------------+ | 1 | Switching to Zig from Rust | I love C | | 2 | RAG in SQLite | AI first database | | 3 | Rewriting SQLite | We are no more a sqlite-fork | | 4 | Offline Writes in SQLite | Lets sync | +----+----------------------------+------------------------------+ sqlite> INSERT INTO author_post(user_id, post_id) VALUES (1, 3), (1, 4); sqlite> INSERT INTO author_post(user_id, post_id) VALUES (3, 1), (3, 2); sqlite> INSERT INTO posts(title, content) VALUES('Limbo', 'SQLite in Rust'); sqlite> INSERT INTO author_post(user_id, post_id) VALUES (3, 5); INSERT INTO author_post(user_id, post_id) VALUES (1, 5); sqlite> SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id; +----+------------------------------+----------------+ | id | post | authors | +----+------------------------------+----------------+ | 1 | I love C | Pekka | | 2 | AI first database | Pekka | | 3 | We are no more a sqlite-fork | Glauber | | 4 | Lets sync | Glauber | | 5 | SQLite in Rust | Pekka, Glauber | +----+------------------------------+----------------+ sqlite> SELECT * FROM author_post; +---------+---------+ | user_id | post_id | +---------+---------+ | 1 | 3 | | 1 | 4 | | 3 | 1 | | 3 | 2 | | 3 | 5 | | 1 | 5 | +---------+---------+ sqlite> SELECT rowid, * FROM author_post; +-------+---------+---------+ | rowid | user_id | post_id | +-------+---------+---------+ | 1 | 1 | 3 | | 2 | 1 | 4 | | 3 | 3 | 1 | | 4 | 3 | 2 | | 5 | 3 | 5 | | 6 | 1 | 5 | +-------+---------+---------+ sqlite> ``` The interesting part here is this `rowid` ```sql SELECT rowid, * FROM author_post; ``` This table `author_post` has primary key identified from the combination of the `user_id` and `post_id` column. However, storing `rowid` makes no sense, as it is just redundant, right? Why would you not want to have the `rowid`? The answer is very naive and subtle, but might impact the database querying if the `author_post` has millions and billions of rows. The rowid space will be wasted for no reason. Hence, we can avoid creating the rowid using the `WITHOUT ROWID` parameter or option while creating the table. ```sql DROP TABLE author_post; CREATE TABLE author_post ( user_id INTEGER NOT NULL, post_id INTEGER NOT NULL, PRIMARY KEY (user_id, post_id), FOREIGN KEY (user_id) REFERENCES users (id), FOREIGN KEY (post_id) REFERENCES posts (id) ) WITHOUT ROWID; ``` The rest of the commands from inserts to selection and all of the things we saw above will remain the same. ```sql -- adding authors/users INSERT INTO users(name) VALUES ('Glauber'), ('Jamie'), ('Pekka'); -- adding posts written by pekka INSERT INTO posts(title, content) VALUES ('Switching to Zig from Rust', 'I love C'); INSERT INTO posts(title, content) VALUES ('RAG in SQLite', 'AI first database'); INSERT INTO author_post(user_id, post_id) VALUES (3, 1), (3, 2); -- adding posts written by glauber INSERT INTO posts(title, content) VALUES ('Rewriting SQLite', 'We are no more a sqlite-fork'); INSERT INTO posts(title, content) VALUES ('Offline Writes in SQLite', 'Lets sync'); INSERT INTO author_post(user_id, post_id) VALUES (1, 3), (1, 4); -- adding a post co-authored by pekka and glauber INSERT INTO posts(title, content) VALUES('Limbo', 'SQLite in Rust'); INSERT INTO author_post(user_id, post_id) VALUES (3, 5); INSERT INTO author_post(user_id, post_id) VALUES (1, 5); ``` However, when you query the `author_post` table, the `rowid` will not be returned as it doesn't exists. ```sql SELECT * FROM author_post; ``` ```sql SELECT rowid, * FROM author_post; ``` This would return error as the `rowid` doesn't exists. We will still need the `rowid` in `posts` and `users` table as those are the `PRIMARY KEY` columns. Both the tables `users` and `posts` will have `rowid` referenced or aliased as id of their respective columns as we know because of the special case of `INTEGER PRIMARY KEY` ```sql SELECT rowid, _rowid_, * FROM users; ``` ```sql SELECT rowid, _rowid_, * FROM posts; ``` Only in the case of `author_post` or any junction table, where the primary key is referenced as a combination of other two foreign keys, we don't need the `rowid` as the junction table doesn't have any unique data in it, it is just the mapping of two existing data points/records in the other two tables in the database.

SQLite SQL: One to One Relations

Fri, 19 Sep 2025 00:00:00 UTC

## One to One Relations I might missed this basic relationship model as I haven't really found it quite commonly used, but its still used in very specific examples. That one relationship is `one-to-one`. As the name suggests, it maps one row to exactly one row. Unlike the `one-to-many` relation which had one of the records/rows from the tables or entities connected by multiple ropes, both the records/rows from the tables or entities are connected by only one rope in `one-to-one` relation. Where do you think this could be used? Since it can't be applied to a user and a posts relation model, as that will be a too rigid constrained relationship. One user can only author one post, and one post can only be authored by one user. So, can you think of a relation where one record or row is tied to exactly one other record or row in the other table? Well, it could be user and his unique card, passport or even his subscriptions. 1. One user can only have one passport (or any other identity document). 2. One user can only have one credit card (for a specific bank) 3. One user can only have one subscription (to a specific service) In those cases, I think the `one-to-one` relation serves well and is maybe the only way to get over the constraint. You can definitely restrict the `one-to-many` relation to get this done, but might be a little wired. Will check that in other post. ## Creating a One to One Relation How do we define a one to one relation, if we add a foreign key to the table that can refer multiple entities, so maybe if that foreign key is the primary key? Ok, let me explain more clearly. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, email TEXT NOT NULL, password TEXT NOT NULL ); CREATE TABLE news_subscriptions ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, price INTEGER NOT NULL, status TEXT NOT NULL, expiry_date TEXT NOT NULL, user_id INTEGER NOT NULL, FOREIGN KEY (user_id) REFERENCES users(id) ); ``` In the above example, we have a `users` table and `news_subscriptions` table, where `news_subscriptions` table is a one to one relation to `users` table. Here, the `user_id` is a foreign key in `news_subscriptions` table that can refer to the `id` of the `users` table. This as you know is a `one-to-many` relation, as one user can have multiple news subscriptions. Becuase here, one user can create multiple subscriptions, we don't want that right? How can we restrict the many subscriptions to one user? What if the `PRIMARY KEY` of the `subscriptions` table was the same as the `users` table? That will restrict it to add any duplicate subscription for the same user. Solved! ```sql DROP TABLE IF EXISTS users; DROP TABLE IF EXISTS news_subscriptions; CREATE TABLE users ( id INTEGER PRIMARY KEY, email TEXT NOT NULL, password TEXT NOT NULL ); CREATE TABLE news_subscriptions ( user_id INTEGER PRIMARY KEY, name TEXT NOT NULL, price INTEGER NOT NULL, status TEXT NOT NULL, expiry_date TEXT NOT NULL, FOREIGN KEY (user_id) REFERENCES users(id) ); ``` Here, instead of a separate `user_id` column, we have made the `user_id` the `PRIMARY KEY` of the `news_subscriptions` table. NOTE: OOPs! don't forget to turn on the `PRAGMA foreign_keys` option. > PRAGMA foreign_keys = ON; This will tie the `news_subscriptions` table directly to an record/row that would require only one record/row in the `users` table. Let's try inserting a few users: ```sql INSERT INTO users(email, password) VALUES('harry', 'harry123'), ('ron', 'ron123'), ('malfoy', 'malfoy123'); ``` This will insert 3 records/rows in the `users` table. ```sql SELECT * FROM users; ``` Now, let's try inserting a subscription: ```sql INSERT INTO news_subscriptions(user_id, name, price, status, expiry_date) VALUES(1, 'Wizardry Weekly', 500, 'active', '2025-12-31'); ``` This will insert 1 record/row in the `news_subscriptions` table. This is a subscription for the user with `id` as `1`. ```sql SELECT * FROM news_subscriptions; ``` Let's get the user info too: ```sql SELECT u.id, u.email, n.* FROM users u JOIN news_subscriptions n ON u.id = n.user_id; ``` Now, if we try to insert another subscription for the same user: ```sql INSERT INTO news_subscriptions(user_id, name, price, status, expiry_date) VALUES(1, 'Muggle Cup', 250, 'active', '2025-12-31'); ``` This will throw an error, as we have already inserted a subscription for the user with `id` as `1`. Hence we successfully created a one to one relation between the `users` and `news_subscriptions` tables. ``` sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, email TEXT NOT NULL, password TEXT NOT NULL ); CREATE TABLE news_subscriptions ( user_id INTEGER PRIMARY KEY, name TEXT NOT NULL, price INTEGER NOT NULL, status TEXT NOT NULL, expiry_date TEXT NOT NULL, FOREIGN KEY (user_id) REFERENCES users(id) ); sqlite> .mode table sqlite> INSERT INTO users(email, password) VALUES('harry', 'harry123'), ('ron', 'ron123'), ('malfoy', 'malfoy123'); sqlite> SELECT * FROM users; +----+--------+-----------+ | id | email | password | +----+--------+-----------+ | 1 | harry | harry123 | | 2 | ron | ron123 | | 3 | malfoy | malfoy123 | +----+--------+-----------+ sqlite> INSERT INTO news_subscriptions(user_id, name, price, status, expiry_date) VALUES(1, 'Wizardry Weekly', 500, 'active', '2025-12-31'); sqlite> SELECT * FROM news_subscriptions; +---------+-----------------+-------+--------+-------------+ | user_id | name | price | status | expiry_date | +---------+-----------------+-------+--------+-------------+ | 1 | Wizardry Weekly | 500 | active | 2025-12-31 | +---------+-----------------+-------+--------+-------------+ sqlite> INSERT INTO news_subscriptions(user_id, name, price, status, expiry_date) VALUES(1, 'Muggle Cup', 250, 'active', '2025-12-31'); Runtime error: UNIQUE constraint failed: news_subscriptions.user_id (19) sqlite> sqlite> SELECT u.id, u.email, n.* FROM users u JOIN news_subscriptions n ON u.id = n.user_id; +----+-------+---------+-----------------+-------+--------+-------------+ | id | email | user_id | name | price | status | expiry_date | +----+-------+---------+-----------------+-------+--------+-------------+ | 1 | harry | 1 | Wizardry Weekly | 500 | active | 2025-12-31 | +----+-------+---------+-----------------+-------+--------+-------------+ sqlite> ``` Dead simple, this is how we create `one-to-one` relation. We define the `primary key` of the child table as the `foreign key` of the parent table.

SQLite SQL: Many to Many Relations

Thu, 18 Sep 2025 00:00:00 UTC

## Many to Many Relations We have seen [one-to-many](https://www.meetgor.com/sqlog/sqlite-one-to-many-foreign-key/) relationship and also the [self-join](https://www.meetgor.com/sqlog/sqlite-self-join-relations) relationship. Let's extend it with `many-to-many` relationship. Here one row can refer to many other rows, but that other row can also be referred by multiple rows. That was confusing, let's take the simple example of `users` and `posts` as usual. - In `One-To-Many` relation, one post can only be written by one user. However, one user can author multiple posts. - In `Many-To-Many` relation, one user can author multiple posts and one post can be written by multiple users. That is the distinction, it is not about the relation, it is more about the design that your system needs. Let's start creating one simple example: ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL ); ``` We have created two tables, `users` and `posts` each representing their own data. However, we haven't referenced the users from the posts table, because that might not work out well. Why? Well, there are few considerations in a `many-to-many` relationship. - In a `one-to-many` relation, we could simply add a `user_id` column in posts. That works because each post has exactly one author. - But in a `many-to-many` relation, a single post can have multiple authors. If we put a `user_id` inside posts, we’d be forced to choose only one user, right? What if we tried multiple columns like `user_id_1`, `user_id_2` and so on? You can see that won't work or rather create a lot of confusion and redundancy. What if a post has 4 authors? Or 6? Not practical but could be an exception. When we say, `many-to-many` we truly have to mean `MANY` posts can be written by `ONE` author and `MANY` authors can write one `post`. This is where we need to create a separate table usually called as the junction or an association table. There are [other](https://en.wikipedia.org/wiki/Associative_entity) names too. For this we can create a table `author_post` (not the best of names but it's usually the table and it's relation as the name). This table will refer the `PRIMARY KEY`s of both the `users` and the `posts` table to map it to a single `post`, and the `PRIMARY KEY` of this table becomes the combinational pair of those two primary keys. ```sql CREATE TABLE author_post ( user_id INTEGER NOT NULL, post_id INTEGER NOT NULL, PRIMARY KEY (user_id, post_id), FOREIGN KEY (user_id) REFERENCES users(id), FOREIGN KEY (post_id) REFERENCES posts(id) ); ``` Here, the `user_id` and `post_id` are the `PRIMARY KEY`s of the `users` and the `posts` table respectively, however those are the `FOREIGN KEY`s for this `author_post` table. The `PRIMARY KEY` of the `author_post` table becomes, the combination of both `user_id` and `post_id` as `PRIMARY KEY(user_id, post_id)`. Let's take a example to make it clear as usual: Let's say `Pekka`(sir) is writing very technical details in a post and `Glauber` (sir) is writing the practical details in a post, both are important.They write individual posts, both of them have their audiences on the same blog. But, there is a topic they want to write together, how can they co-author a `posts` with the previous design of `one-to-many` relation? In the case of `one-to-many` relation, the `users` let's say `Pekka` with `id` as `3` and `Glauber` as `id` with `1`. They wrote their own posts with id `98` and `99` respectively. Now, for the `100` th `post` they want to write it together. If we create a `posts` record with the `author_id` as `3`, then it will just refer the user with the id `3` which is `Pekka`, then the user with `id` as `1` can't be referred to the `posts` record. If we tried to create 2 posts with different `user_id` it might be redundant to create 2 posts, and those might lead to inconsistent number of post counts, since each post will have unique identifier. So, essentially, co-authoring in this `one-to-many` relationship, is not possible, we need to do something to make them write together. The `many-to-many` relation will help them write a `post` together. They both can write the `posts` record with just the `title` and `content`. Let's say the `post` was created with `id` as `100` Them, while publishing the `post`, they can provide the `user_id`s, so here, two records of `author_post` will be created, one with `user_id` as `3`and `post_id` as `100`, similarly another record will be created with `user_id` as `1` and `post_id` as `100`. Notice how we are duplicating the `post_id` (it is same, as `100`), but the `user_id` is different in each case. Hence the constraint is satisfied, the `user_id` and `post_id` the combination of those two is unique. Hence those two can write a `post` together now, the `100` th post is co-authored by `Pekka` and `Glauber`. And finally `Pekka` and `Glauber` decided to write a [post](https://turso.tech/blog/introducing-limbo-a-complete-rewrite-of-sqlite-in-rust) NOTE: This is just a made up example, I found it easier to relate and come up with a example of the similar blog post kind of table, so took that and wrote this, sorry if anyone got hurt. Let's look at actual queries: Create three users `Glauber`, `Jamie` and `Pekka`. ```sql INSERT INTO users(name) VALUES ('Glauber'), ('Jamie'), ('Pekka'); ``` These users are the authors with the ids `1`, `2` and `3` respectively. ```sql SELECT * FROM users; ``` As we can see in this users table. Then, let's Pekka write the posts. He wrote 2 posts, one with the title `Switching to Zig from Rust` and the other with the title `RAG in SQLite`. This is just for fun. ```sql INSERT INTO posts(title, content) VALUES ('Switching to Zig from Rust', 'I love C'); INSERT INTO posts(title, content) VALUES ('RAG in SQLite', 'AI first database'); SELECT * FROM posts; ``` Here, the `id` of the first post is `1` and the `id` of the second post is `2`. So, `Pekka` has authored posts with id `1` and `2` but that is not mapped yet. Let's map it in the `author_post` table, we know that the `PRIMARY KEY` for the `author_post` is a combination of `user_id` or `author_id` and the `post_id`, we know that `Pekka` has a `user_id` of `3` and he has written the `post` with `id`s `1` and `2`, so we'll simply insert those in to the `author_post` table as individual records. ```sql INSERT INTO author_post(user_id, post_id) VALUES (3, 1), (3, 2); ``` This will create two distinct records rightly so, for each posts. Then, Glauber decided to write a couple of posts. Let's say he wrote 2 posts, one with the title `Rewriting SQLite` and the other with the title `Offline Writes in SQLite`. ```sql INSERT INTO posts(title, content) VALUES ('Rewriting SQLite', 'We are no more a sqlite-fork'); INSERT INTO posts(title, content) VALUES ('Offline Writes in SQLite', 'Lets sync'); SELECT * FROM posts; ``` Here, the posts created by `Glauber` have ids `3` and `4`. Now, we need to map them in the `author_post` table. We know that `Glauber` has a `user_id` of `1` and he has written the `post` with `id`s `3` and `4`, so we'll simply insert those in to the `author_post` table as individual records. ```sql INSERT INTO author_post(user_id, post_id) VALUES (1, 3), (1, 4); ``` So, this will create two distinct entries in the `author_post` table for the two posts created by `users` with `id` as `1`. Now, the question of how they co-author a post? That is a piece of cake. Let's say they wrote a post with the title `Limbo`. ```sql INSERT INTO posts(title, content) VALUES('Limbo', 'SQLite in Rust'); ``` Now, this will create a new post with `id` as `5`, let's map it in the `author_post` table ```sql INSERT INTO author_post(user_id, post_id) VALUES (3, 5); INSERT INTO author_post(user_id, post_id) VALUES (1, 5); ``` Here, we create two records but the `post_id` value is the same, the `user_id` value changes for both the posts, one for `Glauber` and one for `Pekka` with ids 1 and 3 respectively. SQueaLicios! Now, you will say, isn't it still doing redundant insertions of two `author_post` records? Well ... yes ... but. Yes, that is true, it looks like we are doing redundant as we will create records in the `author_post` table (let's call it the junction table or the middle or connector table). But there is no redundant data, every row gives us unique insight into what it is actually storing. We aren't storing the original `posts` record twice or multiple twice, we are just `referencing` it with the `user_id` and the `post_id` Another question would be, how the heck do we query this `post` which is co-authored? Or that has multiple authors in it, easy peasy, sqly. Bear with some SQL shenanigans, some joins and wizardry. ```sql SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id HAVING COUNT(u.id) > 1; ``` Don't worry if you don't get this query as a whole, just understand three things. - What are we querying? The post id, the post content, and the author names concatenated with `,` - From where? The `posts`, `users` and the `author_post` table. - How? Querying all the posts, joining the `posts` with the id in the `author_post` table, then also fetching the `name` of the `author` from the `users` name. We are grouping by post id in order to keep it per post, and not duplicate the posts in case of multiple authors, the last filtering is the `HAVING` that is done as a aggregate of the count of users in the `author_post` table having more than one author in the mapping. What about all the posts? We just remove the filter of `HAVING` clause to include all the posts, rather than only posts having more than 1 author. ```sql SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id; ``` The heavy SQLog: ```sql sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, title TEXT NOT NULL, content TEXT NOT NULL ); sqlite> CREATE TABLE author_post ( user_id INTEGER NOT NULL, post_id INTEGER NOT NULL, PRIMARY KEY (user_id, post_id), FOREIGN KEY (user_id) REFERENCES users(id), FOREIGN KEY (post_id) REFERENCES posts(id) ); sqlite> .mode table sqlite> INSERT INTO users(name) VALUES ('Glauber'), ('Jamie'), ('Pekka'); sqlite> INSERT INTO posts(title, content) VALUES ('Switching to Zig from Rust', 'I love C'); INSERT INTO posts(title, content) VALUES ('RAG in SQLite', 'AI first database'); SELECT * FROM posts; +----+----------------------------+-------------------+ | id | title | content | +----+----------------------------+-------------------+ | 1 | Switching to Zig from Rust | I love C | | 2 | RAG in SQLite | AI first database | +----+----------------------------+-------------------+ sqlite> INSERT INTO author_post(user_id, post_id) VALUES (3, 1), (3, 2); sqlite> INSERT INTO posts(title, content) VALUES ('Rewriting SQLite', 'We are no more a sqlite-fork'); INSERT INTO posts(title, content) VALUES ('Offline Writes in SQLite', 'Lets sync'); SELECT * FROM posts; +----+----------------------------+------------------------------+ | id | title | content | +----+----------------------------+------------------------------+ | 1 | Switching to Zig from Rust | I love C | | 2 | RAG in SQLite | AI first database | | 3 | Rewriting SQLite | We are no more a sqlite-fork | | 4 | Offline Writes in SQLite | Lets sync | +----+----------------------------+------------------------------+ sqlite> INSERT INTO author_post(user_id, post_id) VALUES (1, 3), (1, 4); sqlite> INSERT INTO posts(title, content) VALUES('Limbo', 'SQLite in Rust'); sqlite> INSERT INTO author_post(user_id, post_id) VALUES (3, 5); INSERT INTO author_post(user_id, post_id) VALUES (1, 5); sqlite> SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id HAVING COUNT(u.id) > 1; +----+----------------+----------------+ | id | post | authors | +----+----------------+----------------+ | 5 | SQLite in Rust | Pekka, Glauber | +----+----------------+----------------+ sqlite> SELECT * FROM posts; +----+----------------------------+------------------------------+ | id | title | content | +----+----------------------------+------------------------------+ | 1 | Switching to Zig from Rust | I love C | | 2 | RAG in SQLite | AI first database | | 3 | Rewriting SQLite | We are no more a sqlite-fork | | 4 | Offline Writes in SQLite | Lets sync | | 5 | Limbo | SQLite in Rust | +----+----------------------------+------------------------------+ sqlite> SELECT p.id, p.content AS post, GROUP_CONCAT(u.name, ', ') AS authors FROM posts p JOIN author_post up ON p.id = up.post_id JOIN users u ON u.id = up.user_id GROUP BY p.id; +----+------------------------------+----------------+ | id | post | authors | +----+------------------------------+----------------+ | 1 | I love C | Pekka | | 2 | AI first database | Pekka | | 3 | We are no more a sqlite-fork | Glauber | | 4 | Lets sync | Glauber | | 5 | SQLite in Rust | Pekka, Glauber | +----+------------------------------+----------------+ sqlite> ``` That’s the basic idea of a many-to-many relation in SQLite, instead of stuffing multiple IDs into one table, we create a separate mapping table. Each row in that table ties one user to one post. If multiple users are tied to the same post, we just add more mapping rows.

SQLite SQL: Self Join Relations

Wed, 17 Sep 2025 00:00:00 UTC

## Self Join Relations In the last post, we had defined a `one-to-many` relation between the `users` table and the `posts` table. Now, let's create a single table where we can define a `one-to-many` relationship. Yes, within a single table we can reference the primary key and it can act like a link. Since it isn't really about the table, it's more about the row relation form each other, we an connect any rows. Let's take a example of `comments`. ```sql CREATE TABLE comments ( id INTEGER PRIMARY KEY, author TEXT NOT NULL, content TEXT NOT NULL, parent_id REFERENCES comments (id) ); ``` Here, we have created a table `comments`, it has a simple `id` as a `INTEGER PRIMARY KEY` (we know what that means right? check out 2 articles back to find out.). There are `author` and `content` columns that just hold some data, it could be any thing that actually stores the data. The final column is vital as it makes the `comments` table usable. We have defined `parent_id` as a `REFERENCES comments (id)`, this is the key to the `self-join` relationship. It will refer any other comment with its id. So, let's say, you have created a first comment `A`, here we don't have any previous comment, so that can be NULL. However, if someone comments `B` on my comment, then the parent_id will be of `A`. ```sql INSERT INTO comments (id, author, content, parent_id) VALUES (1, 'Harry Potter', 'I think Snape is hiding something.', NULL); INSERT INTO comments (id, author, content, parent_id) VALUES (2, 'Ron Weasley', 'What exactly? ', 1); INSERT INTO comments (id, author, content, parent_id) VALUES (3, 'Draco Malfoy', 'Potter, you just want attention.', 1); INSERT INTO comments (id, author, content, parent_id) VALUES (4, 'Harry Potter', 'Attention is all you need', 3); INSERT INTO comments (id, author, content, parent_id) VALUES (5, 'Ron Weasley', 'Calm down guys', 3); ``` Ok, we have inserted a lot at once, let's break it down - Harry adds the first comment `I think Snape is hiding something.` (parent_id will be NULL, as that is the first comment) - Ron replies to the first comment `What exactly?` (parent_id will be 1, as that is the id of the first comment) - Draco replies to the first comment `Potter, you just want attention.` (parent_id will be 1, as that is the id of the first comment) - Harry replies to Draco's comment `Attention is all you need` (parent_id will be 3, as that is the id of Draco's comment) - Ron replies to Draco's comment `Calm down guys` (parent_id will be 3, as that is the id of Draco's comment) This is really cool, as it makes use of the same id or primary key of the table, referencing some other row. Let's query the data and help you understand better. To get the first comment (base comment), we can use the following query ```sql SELECT id, author, content FROM comments WHERE parent_id IS NULL; ``` To get the replies for a given comment id, we can use the following query ```sql SELECT id, content, parent_id FROM comments WHERE parent_id = 1; ``` To get all the comments for a given author, we can use the following query: ```sql SELECT id, author, content FROM comments WHERE author = 'Harry Potter'; ``` Here's a SQLog: ```sqlite sqlite> CREATE TABLE comments ( id INTEGER PRIMARY KEY, author TEXT NOT NULL, content TEXT NOT NULL, parent_id REFERENCES comments (id) ); sqlite> .mode table sqlite> INSERT INTO comments (id, author, content, parent_id) VALUES (1, 'Harry Potter', 'I think Snape is hiding something.', NULL), (2, 'Ron Weasley', 'What exactly? ', 1), (3, 'Draco Malfoy', 'Potter, you just want attention.', 1), (4, 'Harry Potter', 'Attention is all you need', 3), (5, 'Ron Weasley', 'Calm down guys', 3); sqlite> SELECT * FROM comments; +----+--------------+------------------------------------+-----------+ | id | author | content | parent_id | +----+--------------+------------------------------------+-----------+ | 1 | Harry Potter | I think Snape is hiding something. | | | 2 | Ron Weasley | What exactly? | 1 | | 3 | Draco Malfoy | Potter, you just want attention. | 1 | | 4 | Harry Potter | Attention is all you need | 3 | | 5 | Ron Weasley | Calm down guys | 3 | +----+--------------+------------------------------------+-----------+ sqlite> SELECT id, author, content FROM comments WHERE parent_id IS NULL; +----+--------------+------------------------------------+ | id | author | content | +----+--------------+------------------------------------+ | 1 | Harry Potter | I think Snape is hiding something. | +----+--------------+------------------------------------+ sqlite> SELECT id, content, parent_id FROM comments WHERE author = 'Harry Potter'; +----+------------------------------------+-----------+ | id | content | parent_id | +----+------------------------------------+-----------+ | 1 | I think Snape is hiding something. | | | 4 | Attention is all you need | 3 | +----+------------------------------------+-----------+ sqlite> SELECT id, content, parent_id FROM comments WHERE parent_id = 1; +----+----------------------------------+-----------+ | id | content | parent_id | +----+----------------------------------+-----------+ | 2 | What exactly? | 1 | | 3 | Potter, you just want attention. | 1 | +----+----------------------------------+-----------+ sqlite> ``` As you can see, we have created a `one-to-many` relation with the same single table. Here, one comment can have mulitple comments. This makes it a bit recursive but it is what it is. That's how certain systems are. Self referencing relations!

SQLite SQL: One to Many Relation with Foreign Key

Tue, 16 Sep 2025 00:00:00 UTC

## One to Many Relation with Foreign Key Back to the question that we raised in the [previous](https://www.meetgor.com/sqlog/sqlite-foreign-key-table-constraint/) post, "What would happen if there are more than one row in the primary table that references the same id in the foreign table?" This is what is precisely called the `one-to-many` relation, or what the foreign key is used for. - One row in the foreign(parent/other) table - that can be referenced by many rows in the primary(child) table So, taking the example from the previous post of author and posts, Let's define the schema again: ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ); ``` the analogy can be stated as: - One author can have many posts - One post can have only one author Right? If you look at the schema, the posts table is having the foreign key `author_id` that is referencing the primary key `id` of the `users` table. So the author can be referenced by multiple posts, however one post record can be refer only one author. I mean that is the design, a post can have multiple authors if we wanted to but we then would have to change the design then, we check that in the next post. Let's keep the `Foreign Key` pragma/setting constraint on throughout this post. ```sql PRAGMA foreign_keys=on; ``` Let's insert a few users and posts to get what we mean by the `one-to-many` relation. ```sql INSERT INTO users(name) VALUES('harry'), ('ron'), ('malfoy'); ``` We are inserting 3 users, `harry`, `ron`, and `malfoy` which will have ids as `1`, `2`, `3` respectively. ```sql SELECT * FROM users; ``` As you would see in the output, we have 3 users with ids `1`, `2`, and `3`. Now, let's insert a few posts. ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1), ('Expecto Patronum', 1), ('hi, I am ron', 2), ('hi, I am malfoy', 3); ``` I am inserting the `2nd` post as `harry`'s because it will confuse with the author_id and the `post.id` ```sql SELECT * FROM posts; ``` As you would see in the output, we have 4 posts with ids `1`, `2`, `3`, `4` the first 2 are of `harry`, then the next ron and malfoy respectively. Below are some neat little queries to visualize it better. ```sql SELECT * FROM posts JOIN users ON posts.author_id = users.id; ``` ```sql SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; ``` This is the sqlog for the things that we did so far. ```sqlite sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ); sqlite> PRAGMA foreign_keys=on; sqlite> INSERT INTO users(name) VALUES('harry'), ('ron'), ('malfoy'); sqlite> .mode table sqlite> SELECT * FROM users; +----+--------+ | id | name | +----+--------+ | 1 | harry | | 2 | ron | | 3 | malfoy | +----+--------+ sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1), ('Expecto Patronum', 1), ('hi, I am ron', 2), ('hi, I am malfoy', 3); sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | 2 | | 4 | hi, I am malfoy | 3 | +----+------------------+-----------+ sqlite> SELECT * FROM posts JOIN users ON posts.author_id = users.id; +----+------------------+-----------+----+--------+ | id | content | author_id | id | name | +----+------------------+-----------+----+--------+ | 1 | hi, I am harry | 1 | 1 | harry | | 2 | Expecto Patronum | 1 | 1 | harry | | 3 | hi, I am ron | 2 | 2 | ron | | 4 | hi, I am malfoy | 3 | 3 | malfoy | +----+------------------+-----------+----+--------+ sqlite> SELECT posts.*, users.name AS author_name FROM posts JOIN users ON posts.author_id = users.id; +----+------------------+-----------+-------------+ | id | content | author_id | author_name | +----+------------------+-----------+-------------+ | 1 | hi, I am harry | 1 | harry | | 2 | Expecto Patronum | 1 | harry | | 3 | hi, I am ron | 2 | ron | | 4 | hi, I am malfoy | 3 | malfoy | +----+------------------+-----------+-------------+ sqlite> ``` Now, let's explore this: - Harry has 2 posts, which means one user can author multiple post by design. - One post can be always be author by only one user. Let's assume all the `NULL` values are not allowed for now, we can see one post can only be by one author, that is a constraint of the `FOREIGN KEY`, as we are referencing only one foreign key in the posts table. This is what we call `one-author-many-posts` or `one-to-many` relation. You can also call it `many-to-one` but then you have to change it to `many-posts-one-author` that sounds a little wired. Its basically the same thing but from the different perspective, I prefer saying `one-to-many` since most of the times, the foreign/parent table needs to exist first inorder for the parent/primary table to reference any key from it. One is on the side of the parent table (foreign table) and the Many is usually on the primary or the place where the `foreign key` is placed (in this case it is placed in the posts table). What about multiple foreign keys? Yes, you can have multiple foreign keys, but those will be different keys, you can't have `author_id_1`, `author_id_2` and then refer the same foreign key as `users.id`, that is possible. However, we need to define separate table for mapping users and posts and that will be called as `many-to-many` relations as will check in the next post.

SQLite SQL: Foreign KEY table constraint

Mon, 15 Sep 2025 00:00:00 UTC

## The Foreign KEY Table constraint Foreign Keys are the fundamentals of any relational databases, as they are the ones that make the connection or the relations among the tables in our database system. > Foreign key as the name suggest, this is a key referencing or pointing to a foreign (other) table, and that key could be a primary key for that table, hence referred to as key. So, with a foreign key we can connect the data/records/row from other table to the table in which the foreign key is linked. Think of it like a string(rope) attaching one record to the other. It is a link between two tables. Now, how we define the `FOREIGN KEY` constraint is what the rope will be tied to and how. Let's take a look at how the syntax of defining a `FOREIGN KEY` constraint looks like in SQLite. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ); ``` You will need atleast two tables in order to map a foreign key constraint. In the above example, we have created a table `users` that will form the base table, and the table `posts` is what is linking the `id` from the `users` table to itself. This means one post can link to one user, we are using that reference in the current `posts` table with the alias of `author_id`. This way the `author_id` is the rope (string whatever you prefer saying, I think developers get confused when I say string) that connects a record in the post table to the record in the users table. We can conclude a few things from this: - We need to define what that `author_id` is for, for each record we insert into the `posts` table. - If we don't define the `author_id` then will it be automatically populated? Nope! - The `FOREIGN KEY` at its core is just like any other column, its just that how we set that column is what is important for the linking part. - It depends on how the actual key i.e. the column in the foreign key is defined in the foreign table. Might still not make sense, let me explain with a few example. Let's insert a bunch of users. ```sql INSERT INTO users(name) VALUES('harry'), ('ron'), ('malfoy'); ``` ```sql SELECT * FROM users; ``` This has 3 records each assigned ids from 1 to 3. Now, let's insert a bunch of posts. ```sql INSERT INTO posts(content) VALUES ('hi, I am who?'); ``` ```sql SELECT * FROM posts; ``` That `author_id` is `NULL` because we didn't inserted anythign against it. ``` sqlite> CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL ); CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ); sqlite> INSERT INTO users(name) VALUES('harry'), ('ron'), ('malfoy'); sqlite> .mode table sqlite> SELECT * FROM users; +----+--------+ | id | name | +----+--------+ | 1 | harry | | 2 | ron | | 3 | malfoy | +----+--------+ sqlite> INSERT INTO posts(content) VALUES ('hi, I am who?'); sqlite> SELECT * FROM posts; +----+---------------+-----------+ | id | content | author_id | +----+---------------+-----------+ | 1 | hi, I am who? | | +----+---------------+-----------+ sqlite> INSERT INTO users(name, id) VALUES('neville', NULL); sqlite> SELECT * FROM posts; +----+---------------+-----------+ | id | content | author_id | +----+---------------+-----------+ | 1 | hi, I am who? | | +----+---------------+-----------+ sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 3 | malfoy | | 4 | neville | +----+---------+ sqlite> ``` We need to be careful while inserting anything in that `author_id` column, becuase think for a moment: What do we insert in the `author_id` column? Well, we need to have a valid `users(id)` right, that is the id in the users table, that is what we are referencing by that `FOREIGN KEY` constraint. There are other questions that should arise now, like: - What happens if we link something that doesn't exist in the foreign table? (in this case the `users` table) - What happens if we what we have linked is deleted or changed in the foreign table? - What happend if multiple rows refer the same id in the foreign table? The `FOREIGN KEY` constraint will make sure we are linking the valid id and hence if we try to insert any invalid (non-existent) id of the foreign table (here the `users` table) we will fail this constraint and hence the record/row won't be inserted or updated. As I said, this is a rope, the one end of the rope is always the current table record and the other one is what you attach it when inserting or updating a record. So, for starters, we'll insert the users with the id that exist in the first place. In our case, we have 4 authors created with the id from 1 to 4 and names as `harry`, `ron`, `malfoy` and `neville` respectively. Now, let's insert a post with the `author_id` as 1, that is the id of the user with the name `harry` which will be the author of the post. ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); ``` This will set the `author_id` column of the `posts` table to 1, which is the id of the `harry` user. If you want to see some better results, just run some of these to get it clear: ```sql SELECT * FROM posts; ``` Get both tables data: ```sql SELECT * FROM posts JOIN users ON author_id = id; ``` Make column names clear and remove redundant columns: ```sql SELECT posts.*, users.name FROM posts JOIN users ON posts.author_id = users.id; ``` ``` sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); sqlite> SELECT * FROM posts; +----+----------------+-----------+ | id | content | author_id | +----+----------------+-----------+ | 1 | hi, I am who? | | | 2 | hi, I am harry | 1 | +----+----------------+-----------+ sqlite> SELECT * FROM posts JOIN users ON author_id = id; Parse error: ambiguous column name: id sqlite> SELECT * FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+----+-------+ | id | content | author_id | id | name | +----+----------------+-----------+----+-------+ | 2 | hi, I am harry | 1 | 1 | harry | +----+----------------+-----------+----+-------+ sqlite> SELECT posts.*, users.name FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+-------+ | id | content | author_id | name | +----+----------------+-----------+-------+ | 2 | hi, I am harry | 1 | harry | +----+----------------+-----------+-------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+--------+ | id | content | author_id | author | +----+----------------+-----------+--------+ | 2 | hi, I am harry | 1 | harry | +----+----------------+-----------+--------+ sqlite> ``` Now, try inserting a post with the `author_id` which doesn't exist in the `users` table i.e. the id of the users above 4. ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); ``` That inserted successfully? How can that be allowed? The user with id `5` doesn't exist and we are referencing the author (users) that doesn't exist. ## Pragma Foreign keys Well, we have been lied to all along, the SQLite is so flexible by default, you need to nudge it inorder to be a little strict: We need to enable the foreign key constraint checking by setting the `PRAGMA foreign_keys=on;` ``` PRAGMA foreign_keys=on; ``` This setting is off by default due to backward compatibility reasons. You can check if the foreign key constraint checking is enabled by running the following command: ``` PRAGMA foreign_keys; ``` If it returns `1` then its enabled, or `0` if not. Now, now if we run the query where the author id is `5` which means the `user` referenced by that id doesn't exist, we will get the following error: ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); ``` > Runtime error: FOREIGN KEY constraint failed Exactly as we expected. ``` sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+--------+ | id | content | author_id | author | +----+----------------+-----------+--------+ | 2 | hi, I am harry | 1 | harry | +----+----------------+-----------+--------+ sqlite> SELECT * FROM posts; +----+-------------------+-----------+ | id | content | author_id | +----+-------------------+-----------+ | 1 | hi, I am who? | | | 2 | hi, I am harry | 1 | | 3 | hi, I am hermoine | 5 | +----+-------------------+-----------+ sqlite> PRAGMA foreign_keys=on ...> ; sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); Runtime error: FOREIGN KEY constraint failed (19) sqlite> ``` However if we insert the `posts` record with a valid `author_id` then: ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` This will readily insert the record, since the `author_id` of 2 exists in the `users` table. ```sql SELECT * FROM posts; ``` Neat. ``` sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+-------------------+-----------+ | id | content | author_id | +----+-------------------+-----------+ | 1 | hi, I am who? | | | 2 | hi, I am harry | 1 | | 3 | hi, I am hermoine | 5 | | 4 | hi, I am ron | 2 | +----+-------------------+-----------+ sqlite> ``` Now, back to those three questions: - What happens if we link something that doesn't exist in the foreign table? (in this case the `users` table) - What happens if we what we have linked is deleted or changed in the foreign table? - What happend if multiple rows refer the same id in the foreign table? Let's tackle one by one > NOTE: We'll assume PRAGMA foreign_keys=on; from here on ## Invalid Foreign Key We already covered this in the previous discussion, if we link something that doesn't exist in the foreign table (in this case the `users` table) then we will fail the `FOREIGN KEY` constraint and hence the record/row won't be inserted or updated. But for this to fail we need to make sure the `PRAGMA foreign_keys` is on or toggled on. ## Deleted or Updated Foreign Key What happens, if we inserted a record to the primary table (here the `posts` table) the `author_id` was existing and valid at the time of insertion, however after a while the `user` got deleted, what happens to the record that the `posts` record/row that still references the deleted user id that is the `author_id` alias. Enter `ON DELETE` and `ON UPDATE` clause on the `FOREIGN KEY` constraint. It turns out, we can define the behaviour of the `FOREIGN KEY` constraint when the referenced row is deleted or updated in the foreign table (here the `users` table). So, we need to define, what happens to the record/row in the primary table (here the `posts` table) when the referenced row in the foreign table (here the `users` table) is deleted or updated. There are 5 options to choose from: 1. `NO ACTION`: Do nothing (default) 2. `RESTRICT`: Stop the foreign key table from deleting the record/key in the foreign table 3. `SET NULL`: Set the primary table's record foreign key to `NULL` 4. `SET DEFAULT`: Set the primary table's record foreign key to the default value. 5. `CASCADE`: Delete the primary table's record/row if the referenced row in the foreign table is deleted. Phew! This opens a lot of options to play with: We already know the `NO ACTION`, it will just let it happen, nothing will be done. It's not recommended though, but SQLite is flexible, how many times I have to say it? Double edged sword. ### Restrict Let's drop the `posts` table and start a fresh with the `ON DELETE` and `ON UPDATE` as `RESTRICT` option on the `FOREIGN KEY` constraint. #### Restrict on delete We can restrict the deletion of the foreign records referred by the primary record. We use the `ON DELETE RESTRICT` as the option on the `FOREIGN KEY` constraint. ```sql DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE RESTRICT ); ``` This will now restrict the deletion of the foreign records referred by the primary record. In this case, if a record/row in `users` table which is refereced in the `posts` table is deleted, it will prevent it, it won't be deleted. However, any other record in the `users` table can be deleted. Let's first insert a few rows into the `posts` table since we dropped it. ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` These are valid records, since the `author_id` of 1 and 2 exist in the `users` table. However the following query will fail, since the `author_id` of 5 doesn't exist in the `users` table. We already know this: ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); ``` Now, Let's look at all the `posts` and their authors ```sql SELECT * FROM posts; ``` This is the list of `users` that we have: ```sql SELECT * FROM users; ``` Now, let's try deleting a record in the `users` table. We know that `users` with id `1` and `2` are referenced in the `posts` table. ```sql DELETE FROM users WHERE id = 2; ``` This will fail, since we are trying to delete a record that is referenced in the `posts` table. We can check if the `users` with `id` 2 is still there: ```sql SELECT * FROM users; ``` It is indeed ther, it restricted the deletion. However if we delete the `users` where id is `3` which is not referenced in the `posts` table, it will succeed. ```sql DELETE FROM users WHERE id = 3; ``` This deleted the `users` with `id` 3. ```sql SELECT * FROM users; ``` As you can see the dirt `malfoy` was deleted. ``` sqlite> DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE RESTRICT ); sqlite> SELECT * FROM posts; sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+----------------+-----------+ | id | content | author_id | +----+----------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | hi, I am ron | 2 | +----+----------------+-----------+ sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am hermoine', 5); Runtime error: FOREIGN KEY constraint failed (19) sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 3 | malfoy | | 4 | neville | +----+---------+ sqlite> DELETE FROM users where id = 2; Runtime error: FOREIGN KEY constraint failed (19) sqlite> DELETE FROM users where id = 3; sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 4 | neville | +----+---------+ sqlite> ``` What about updates? #### Restrict on update What if we update the users record will it restrict it, nope, unless we set `ON UPDATE` to `RESTRICT`: > This option of `UPDATE ON` is usually not needed, as this is referring to the `UPDATE ON` the `FOREIGN KEY` and not on the entire foreign table columns, so only if you update the foreign key it will restrict or any action will be performed. Which means, if you update the `name` of the `users` table of a record which is referenced in the `posts` table, it will allow it, however when you update the `users` id (which doen't happen unless you've skewed up or it's by design), it will prevent that updation. Note, updating foreign keys which are usually the primary keys of the foreign table is very rare, and not done usually, as it might corrupt the existing data. So, unless you have a great usecase for updating the foreign key of a table, `UPDATE ON` is never used. ```sql DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE RESTRICT ON UPDATE RESTRICT ); ``` Here, we are adding the `ON UPDATE RESTRICT` option on the `FOREIGN KEY` constraint, the `ON DELETE RESTRICT` option won't affect the update option, so it's only triggered when we try to delete the record (hard delete). Let's add few more rows as usual: ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` Now, let's try updating the `users` table: ```sql UPDATE users SET name = 'harry potter' WHERE id = 1; ``` This is allowed, nothing got restricted, as we didn't update the `id` in the `users` table which is a foreign key in the `posts` table, unless we update the `id` of the `users` table, the `update` option will allow any updates of the columns in the foreign table (users table) ```sql SELECT * FROM users; ``` This readily updated the row with `id` 1 to `harry potter` as it didn't mutate/change/update the foreign key `id` in the `users` table which is referenced as `author_id` in the `posts` table. However, if we try to update the `id` of the `users` table, it will fail. ```sql UPDATE users SET id = 3 WHERE id = 2; ``` We are updating the id of the `users` table to 3 which is not the id of the `users` table referenced in the `posts` table, so it will fail. The existing row of id = 2 is `ron`, hence it already has a posts entry which makes it a reference, as we have added a constraint to restrict on updation of the id the foreign key. Now, let's try to update the `id` of the `users` table to the id of the `users` table not referenced in the `posts` table ```sql UPDATE users SET id = 5 WHERE id = 4; ``` The row with id = 4 in the `users` table is `neville` which doesn't have any posts entry, so it will succeed. This will succeed, as we are updating the `id` of the `users` table to the id of the `users` table not referenced in the `posts` table ``` sqlite> DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE RESTRICT ON UPDATE RESTRICT ); sqlite> SELECT * FROM posts; sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+----------------+-----------+ | id | content | author_id | +----+----------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | hi, I am ron | 2 | +----+----------------+-----------+ sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 4 | neville | +----+---------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+--------+ | id | content | author_id | author | +----+----------------+-----------+--------+ | 1 | hi, I am harry | 1 | harry | | 2 | hi, I am ron | 2 | ron | +----+----------------+-----------+--------+ sqlite> UPDATE users SET name = 'harry potter' where id = 1; sqlite> SELECT * FROM users; +----+--------------+ | id | name | +----+--------------+ | 1 | harry potter | | 2 | ron | | 4 | neville | +----+--------------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+----------------+-----------+--------------+ | id | content | author_id | author | +----+----------------+-----------+--------------+ | 1 | hi, I am harry | 1 | harry potter | | 2 | hi, I am ron | 2 | ron | +----+----------------+-----------+--------------+ sqlite> UPDATE users SET name = 'harry' where id = 1; sqlite> UPDATE users SET id = 3 where id = 2; Runtime error: FOREIGN KEY constraint failed (19) sqlite> UPDATE users SET id = 5 where id = 4; sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 5 | neville | +----+---------+ sqlite> ``` ### Set NULL For the rest of this post, we'll only be looking at the `ON DELETE` constraint, the `ON UPDATE` is very rarely used and we have explored already when it is used. In this option, when the record in the foreign table is deleted, the foreign key reference of the record in the primary table will be set to `NULL`. For our example, if we set `ON DELETE SET NULL` to the `FOREIGN KEY` constraint, when the `users` record with id = 2 is deleted, the `author_id` of the `posts` record with id = 3 will be set to `NULL`. ```sql DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE SET NULL ); ``` Now, let's add few more rows as usual: NOTE: I have added one more post for harry to avoid the 1-1 mapping of posts and users in the data ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` Let's take a look at the posts table. ```sql SELECT * FROM posts; ``` Now, let's delete the `users` record with id = 2: ```sql DELETE FROM users WHERE id = 2; ``` This will allow and should, as there is no restriction on the `posts` table, the `FOREIGN KEY` constraint is `SET NULL` when the reference in the `users` table is deleted. ```sql SELECT * FROM posts; ``` As you can see, the `author_id` of the `posts` record with id = 3 has been set to `NULL` ``` sqlite> DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE SET NULL ); sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | 2 | +----+------------------+-----------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+------------------+-----------+--------+ | id | content | author_id | author | +----+------------------+-----------+--------+ | 1 | hi, I am harry | 1 | harry | | 2 | Expecto Patronum | 1 | harry | | 3 | hi, I am ron | 2 | ron | +----+------------------+-----------+--------+ sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 5 | neville | +----+---------+ sqlite> DELETE FROM users WHERE id = 2; sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | | +----+------------------+-----------+ sqlite> ``` ### Set Default In this option, when the record in the foreign table is deleted, the foreign key reference of the record in the primary table will be set to the default value of the foreign key. NOTE: We have not set the default value for the `author_id` column in the `posts` table, so it will be set to `NULL` by default. We can add normal constraints like `DEFAULT`, `UNIQUE`, `NOT NULL`, `CHECK`, etc on the foreign key as and when required. ```sql DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER DEFAULT 1, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE SET DEFAULT ); ``` We are setting default foreign key as `1` which means the `users` record with id = 1 will be the default reference for the `posts` table. If you don't provide the `author_id`, all post will be authored by `harry` what a funny quirk of the posts. Let's add rows as usual: ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` Now, let's take a look at the posts table: ```sql SELECT * FROM posts; ``` We can now try deleting the `users` record with id = 2, Since it is already tied / referenced by the `posts` table, the users record will be deleted, but the `posts` record with `author_id` 2 will be set to the default value `1` which is the id of the `users` record with name `harry`. ```sql DELETE FROM users WHERE id = 2; ``` It ran successfully, but the `posts` record with `author_id` 2 will be set to the default value `1` which is the id of the `users` record with name `harry`. ```sql SELECT * FROM posts; ``` Let's view more closely: ```sql SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; ``` View the SQLog :) ``` sqlite> DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER DEFAULT 1, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE SET DEFAULT ); sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 5 | neville | +----+---------+ sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | 2 | +----+------------------+-----------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+------------------+-----------+--------+ | id | content | author_id | author | +----+------------------+-----------+--------+ | 1 | hi, I am harry | 1 | harry | | 2 | Expecto Patronum | 1 | harry | | 3 | hi, I am ron | 2 | ron | +----+------------------+-----------+--------+ sqlite> DELETE FROM users where id = 2; sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | 1 | +----+------------------+-----------+ sqlite> SELECT posts.*, users.name AS author FROM posts JOIN users ON posts.author_id = users.id; +----+------------------+-----------+--------+ | id | content | author_id | author | +----+------------------+-----------+--------+ | 1 | hi, I am harry | 1 | harry | | 2 | Expecto Patronum | 1 | harry | | 3 | hi, I am ron | 1 | harry | +----+------------------+-----------+--------+ sqlite> ``` Let's take a look at the `CASCADE` option ### Cascade If we set `ON DELETE CASCADE` to the `FOREIGN KEY` constraint, when the `users` record with id = 2 is deleted, the `posts` record with `author_id` 2 will be deleted as well. We can set the `CASCADE` option to the `FOREIGN KEY` constraint as follows: ```sql DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE CASCADE ); ``` Now, let's add few more rows as usual ```sql INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); ``` We now have 3 posts, associated with 2 users. ```sql SELECT * FROM posts; ``` Now, let's delete the `users` record with id = 2 ```sql DELETE FROM users WHERE id = 2; ``` This will not only delete the `users` record with id = 2, but it will also delete the `posts` record with `author_id` 2. ```sql SELECT * FROM posts; ``` Cascade as the name suggest, when the foreign key in the foreign table is deleted, the primary row is cascaded or deleted. ```sql sqlite> DROP TABLE posts; CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT NOT NULL, author_id INTEGER, FOREIGN KEY (author_id) REFERENCES users (id) ON DELETE CASCADE ); sqlite> SELECT * FROM users; +----+---------+ | id | name | +----+---------+ | 1 | harry | | 2 | ron | | 5 | neville | +----+---------+ sqlite> INSERT INTO posts(content, author_id) VALUES ('hi, I am harry', 1); INSERT INTO posts(content, author_id) VALUES ('Expecto Patronum', 1); INSERT INTO posts(content, author_id) VALUES ('hi, I am ron', 2); sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | | 3 | hi, I am ron | 2 | +----+------------------+-----------+ sqlite> DELETE FROM users where id = 2; sqlite> SELECT * FROM posts; +----+------------------+-----------+ | id | content | author_id | +----+------------------+-----------+ | 1 | hi, I am harry | 1 | | 2 | Expecto Patronum | 1 | +----+------------------+-----------+ sqlite> ``` ## Many Foreign Keys What would happen if there are more than one row in the primary table that references the same id in the foreign table? That is a question for another post.

SQLite SQL: PRIMARY KEY table constraint

Sun, 14 Sep 2025 00:00:00 UTC

## The PRIMARY KEY Table constraint The PRIMARY KEY constraint is not actually a column constraint, it is a table constraint. In the previous section we learnt about how to handle single column-level `PRIMARY KEY`. In this section we'll understand how to use `PRIMARY KEY` as table constraint, with that we can use multiple columns to combine the key. Since, `PRIMARY KEY` is a table level constraint, it only can be defined once for the table, as it needs to be unique for each record inserted in that table. So, we can define it with the column or define it at the end, doesn't matter, but it needs to be defined only once. ```sql CREATE TABLE users ( user_id INTEGER PRIMARY KEY, username TEXT NOT NULL, email TEXT NOT NULL ); ``` This is equivalent to: ``` CREATE TABLE users ( user_id INTEGER, username TEXT NOT NULL, email TEXT NOT NULL, PRIMARY KEY (user_id) ); ``` Both of these are setting the column `user_id` as the `PRIMARY KEY` for the table `users`. This will do the same thing, we saw the first one in the previous blog post, in this we will do something different. ## Multiple column PRIMARY KEY The schem design looks a little fragile, we can have `username` and `email` as duplicates right? But we have `user_id` as unique. Rather, we want a combination of the `username` and `email` as unique, do we care about the `user_id` as the `PRIMARY KEY`? It depends: - If your application logic only wants to use `username` and `email` as unique, then the `user_id` is not required. - If your applicaation logic only requires the `user_id` then it doesn't matter if the `username` of `email` are duplicated, the `user_id` will be unique. So, assuming the 1st scenario, let's change the users table schema: ```sql CREATE TABLE users ( username TEXT NOT NULL, email TEXT NOT NULL, PRIMARY KEY (username, email) ); ``` Here we have defined the `username` and `email` as `PRIMARY KEY` for the table `users`. This will take a combination of the `username` and the `email` coluns, and add a `UNIQUE` constraint to it, here there is one more addition that we did which is `NOT NULL` since both the columns are `TEXT` we need to add teh `NOT NULL` constraint, for `INTEGER` and `PRIMARY KEY` it will be handled for us. Let's try inserting a few default values: ```sql INSERT INTO users DEFAULT VALUES; ``` This won't work as the `username` and `email` columns are `NULL` and there is no `NOT NULL` constraint defined on them. Let's try inserting actual values: ```sql INSERT INTO users(username, email) VALUES('john', 'john@abc.com'); ``` This successfully inserts the row into the table. And if we try to insert it again, the duplicate combination of `username` and `email` will throw an error: ```sql INSERT INTO users(username, email) VALUES('john', 'john@abc.com'); ``` So, we either need the `username` or `email` to be unique, but not necessarily both. ```sql INSERT INTO users(username, email) VALUES('john', 'john_new@abc.com'); ``` This successfully inserts the row into the table, as the `username` is same but `email` is unique. Since the PRIMARY key is a combination of `username` and `email` we can allow the combinational unique values into a new record. ```sql SELECT * FROM users; ``` Here's the SQLog :) ``` sqlite> CREATE TABLE users ( username TEXT NOT NULL, email TEXT NOT NULL, PRIMARY KEY (username, email) ); sqlite> INSERT INTO users DEFAULT VALUES; Runtime error: NOT NULL constraint failed: users.username (19) sqlite> INSERT INTO users(username, email) VALUES('john', 'john@abc.com'); sqlite> SELECT * FROM users; +----------+--------------+ | username | email | +----------+--------------+ | john | john@abc.com | +----------+--------------+ sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | john | john@abc.com | +-------+----------+--------------+ sqlite> INSERT INTO users(username, email) VALUES('john', 'john@abc.com'); Runtime error: UNIQUE constraint failed: users.username, users.email (19) sqlite> INSERT INTO users(username, email) VALUES('john', 'john_new@abc.com'); sqlite> SELECT * FROM users; +----------+------------------+ | username | email | +----------+------------------+ | john | john@abc.com | | john | john_new@abc.com | +----------+------------------+ sqlite> SELECT rowid, * FROM users; +-------+----------+------------------+ | rowid | username | email | +-------+----------+------------------+ | 1 | john | john@abc.com | | 2 | john | john_new@abc.com | +-------+----------+------------------+ sqlite> ``` I think that is all we need about `PRIMARY KEY` for the moment, if you need any specific example, we will be exploring `AUTOINCREMENT`, `NOT NULL` cases for `PRIMARY KEY`s and other schema design principles in separate sections.

SQLite SQL: PRIMARY KEY column constraint

Sat, 13 Sep 2025 00:00:00 UTC

## The PRIMARY KEY column constraint The PRIMARY KEY constraint is not actually a column constraint, it is a table constraint. But in this section, we'll only learn about how to handle single column-level `PRIMARY KEY`. The [PRIMARY KEY](https://sqlite.org/lang_createtable.html#the_primary_key) column constraint is a constraint that ensures that a column contains unique values and is the `PRIMARY` way to distinguish between all the rows of that table. From the documentation: > Each table in SQLite may have at most one PRIMARY KEY. If the keywords PRIMARY KEY are added to a column definition, then the primary key for the table consists of that single column. Or, if a PRIMARY KEY clause is specified as a table-constraint, then the primary key of the table consists of the list of columns specified as part of the PRIMARY KEY clause. The PRIMARY KEY clause must contain only column names — the use of expressions in an indexed-column of a PRIMARY KEY is not supported. An error is raised if more than one PRIMARY KEY clause appears in a CREATE TABLE statement. The PRIMARY KEY is optional for ordinary tables but is required for WITHOUT ROWID tables. > If a table has a single column primary key and the declared type of that column is "INTEGER" and the table is not a WITHOUT ROWID table, then the column is known as an INTEGER PRIMARY KEY. See below for a description of the special properties and behaviors associated with an INTEGER PRIMARY KEY. > Each row in a table with a primary key must have a unique combination of values in its primary key columns. For the purposes of determining the uniqueness of primary key values, NULL values are considered distinct from all other values, including other NULLs. If an INSERT or UPDATE statement attempts to modify the table content so that two or more rows have identical primary key values, that is a constraint violation. > According to the SQL standard, PRIMARY KEY should always imply NOT NULL. Unfortunately, due to a bug in some early versions, this is not the case in SQLite. Unless the column is an INTEGER PRIMARY KEY or the table is a WITHOUT ROWID table or a STRICT table or the column is declared NOT NULL, SQLite allows NULL values in a PRIMARY KEY column. SQLite could be fixed to conform to the standard, but doing so might break legacy applications. Hence, it has been decided to merely document the fact that SQLite allows NULLs in most PRIMARY KEY columns. Here is what I think in short: - Used to identify the rows of a table. Hence must be unique - At most one primary key per table (default rowid, multiple primary key not allowed) - Primary key can be a combination of multiple columns or even single column - Only column names allowed (not expressions) - Required for WITHOUT ROWID - INTEGER PRIMARY KEY special behavior - NULL behavior (SQLite quirk vs SQL standard) ## Creating a simple table By default, if you don't create a column in a table as a `PRIMARY KEY`, it is automatically assumed that `rowid` will be used as the unique identifier for that table. Let's create a simple table with columns like `username` and `email`, non of which are `PRIMARY KEY` explicitly mentioned. ```sql CREATE TABLE users ( username TEXT NOT NULL, email TEXT NOT NULL ); ``` Now, let's insert a few columns and check what we can observe. ```sql INSERT INTO users(username, email) VALUES('jim', 'jim@abc.com'); INSERT INTO users(username, email) VALUES('jack', 'jack@abc.com'); SELECT * FROM users; ``` As, you can see, we have populated the table with `username` and `email` columns. But wait, there should be `rowid` right? It is there but its a hidden column (do not touch it). ```sql SELECT rowid, * FROM users; ``` You can see the `rowid` column in the output. ``` sqlite> CREATE TABLE users ( username TEXT NOT NULL, email TEXT NOT NULL ); sqlite> .mode table sqlite> INSERT INTO users(username, email) VALUES('jim', 'jim@abc.com'); sqlite> INSERT INTO users(username, email) VALUES('jack', 'jack@abc.com'); sqlite> SELECT * FROM users; +----------+--------------+ | username | email | +----------+--------------+ | jim | jim@abc.com | | jack | jack@abc.com | +----------+--------------+ sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 2 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> ``` The rowid is used as a primary key for the table, this means it will take the max of the current state of the table and use the next number as its rowid. We can verify that if we try to insert a duplicate value/record/row in the `users` table: Let's insert the `jack` as the user and check if it inserts it, as it would have errored if the username or the email was the primary key, as this value is duplicated. But the `username` and `email` are not the `PRIMARY KEY` in this table, hence it will happily insert it. ```sql INSERT INTO users(username, email) VALUES('jack', 'jack@abc.com'); SELECT rowid, * FROM users; ``` As, you can see it inserted the value and the `rowid` is now `3`. ``` sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 2 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> INSERT INTO users(username, email) VALUES('jack', 'jack@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 2 | jack | jack@abc.com | | 3 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> ``` What would happen if the middle values are deleted? let's delete the row with `rowid` `2`: ```sql DELETE FROM users WHERE rowid = 2; ``` ```sql SELECT rowid, * FROM users; ``` As, you can see, the `rowid` `2` is deleted. The `rowid` will still take the max of the current state of the table, and will use the next number as the `rowid` which in this state should be `4`. Let's insert a new record and check if the `rowid` is the max of the current state of the table. ```sql INSERT INTO users(username, email) VALUES('jill', 'jill@abc.com'); ``` ```sql SELECT rowid, * FROM users; ``` As you can see, the `rowid` is now `4`. Let's delete that and insert one more. ```sql DELETE FROM users WHERE rowid = 4; ``` ```sql SELECT rowid, * FROM users; ``` Now, that the max row of the table is `3` but we had `4` earlier, we need to check what could be the next rowid? ```sql INSERT INTO users(username, email) VALUES('jill', 'jill@abc.com'); SELECT rowid, * FROM users; ``` Ok, as you can see, the `rowid` is now `4`. It is reusing the previous `rowid` value, which might be an issue, very rarely but this is a expected behaviour. Which gives us the answer that there is maybe a quirk or a limitation of `rowid` that it can insert values which are max of the rowid in the current state. ``` sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 2 | jack | jack@abc.com | | 3 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> DELETE FROM users WHERE rowid = 2; sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 3 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> INSERT INTO users(username, email) VALUES('jill', 'jill@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 3 | jack | jack@abc.com | | 4 | jill | jill@abc.com | +-------+----------+--------------+ sqlite> DELETE FROM users WHERE rowid = 4; sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 3 | jack | jack@abc.com | +-------+----------+--------------+ sqlite> INSERT INTO users(username, email) VALUES('joe', 'joe@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+--------------+ | rowid | username | email | +-------+----------+--------------+ | 1 | jim | jim@abc.com | | 3 | jack | jack@abc.com | | 4 | joe | joe@abc.com | +-------+----------+--------------+ sqlite> ``` Ok, that was a wired tangent, but worth noting. We will check how to fix it with the `AUTOINCREMENT` clause in the future posts. Now, let's create a `PRIMARY KEY` column that we can safely call a primary key. ## Creating a column as primary key To create a column as a `PRIMARY KEY` we need to simply add the `PRIMARY KEY` constraint to the column. Let's create a simple table with columns like `username` and `email`, and here `email` is the primary key. ```sql CREATE TABLE users ( username TEXT NOT NULL, email TEXT PRIMARY KEY ); ``` Mentioning `PRIMARY KEY` with email means that each record needs to have a unique `email`. If you attach a `PRIMARY KEY` to a column, the `UNIQUE` constraint is also added automatically. So, saying `email TEXT PRIMARY KEY UNIQUE` is redundant. Let's try inserting default values to the users. ```sql INSERT INTO users DEFAULT VALUES; ``` As you can see, this will fail as the `username` is a `NOT NULL` column. OK, let's just satisfy that `NOT NULL` constraint for username. ```sql INSERT INTO users(username) VALUES('james'); ``` OK, that inserted the value. ```sql SELECT * FROM users; ``` But, the email is empty! Becuase we didn't specify it? Yes, it is a text field, and a primary key, so you need to handle that. What about the rowid? ```sql SELECT rowid, * FROM users; ``` As you can see, the rowid is `1` and email is empty. That's worth noting that `rowid` will be populated independently of the `PRIMARY KEY` column. But if speicifed as `WITHOUT ROWID` in the table creation, it won't be there. Now, let's insert the same record again. We have said the `email` is a `PRIMARY KEY`, so it will be unique right? RIGHT? ```sql INSERT INTO users(username) VALUES('james'); ``` Surprisingly, it inserted that too. ```sql SELECT * FROM users; ``` The email is empty for both the rows, but the username is same, so the `email` is NULL, and you guessed it right, `NULL` is identified as `UNIQUE`. That is a bad schema design. The users table had a text key as a primary key, and we need to add `NOT NULL` to the `PRIMARY KEY` column. ``` sqlite> CREATE TABLE users ( username TEXT NOT NULL, email TEXT PRIMARY KEY ); sqlite> INSERT INTO users DEFAULT VALUES; Runtime error: NOT NULL constraint failed: users.username (19) sqlite> INSERT INTO users(username) VALUES('james'); sqlite> SELECT * FROM users; +----------+-------+ | username | email | +----------+-------+ | james | | +----------+-------+ sqlite> INSERT INTO users(username) VALUES('james'); sqlite> SELECT * FROM users; +----------+-------+ | username | email | +----------+-------+ | james | | | james | | +----------+-------+ sqlite> INSERT INTO users(username) VALUES('jack'); sqlite> SELECT * FROM users; +----------+-------+ | username | email | +----------+-------+ | james | | | james | | | jack | | +----------+-------+ sqlite> ``` > NOTE: NOT NULL is only added automatically for `INTEGER PRIMARY KEY` column, which means the column becomes an alias for the `rowid` table. ## PRIMARY KEY with NOT NULL It turns out that `NOT NULL` is only added automatically for `INTEGER PRIMARY KEY` column, so for rest of the combination of `PRIMARY KEY` columns we need to explicitly add the `NOT NULL` constraint. So, let's add that: ```sql DROP TABLE users; CREATE TABLE users ( username TEXT NOT NULL, email TEXT PRIMARY KEY NOT NULL ); ``` Now, let's try inserting default values to the users. ```sql INSERT INTO users DEFAULT VALUES; ``` As you can see, this will fail as the `username` is a `NOT NULL` column. As we knew in the first attempt. ```sql INSERT INTO users(username) VALUES('james'); ``` OK, that is what we expected, it should fail, as have not specified the `email` column, as that is a primary key and it cannot be null. I mean it could be if we wanted to, but that is not expected to behave. ```sql INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); ``` Ok, this should insert it successfully. ```sql SELECT rowid, * FROM users; ``` As you can see, we have populated the table with `username` and `email` columns. This is the perfect thing to setup. The `rowid` will still be there as discussed, the `email` now is the `PRIMARY KEY` which means any duplicate string email will be constraint violation. ```sql INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); ``` As you can see, this will fail, as we have already inserted a record with the same email. This is expected as we noted that the `PRIMARY KEY` automatically adds the `UNIQUE` constraint to the column. Similary, we can check a few other things. Like if we try to insert a record with the same username but different email: ```sql INSERT INTO users(username, email) VALUES('james', 'james_new@abc.com'); ``` This will succeed, as we have not specified the `username` as primary key, and it is ok to not have unique values for it. ``` sqlite> CREATE TABLE users ( username TEXT NOT NULL, email TEXT PRIMARY KEY NOT NULL ); sqlite> INSERT INTO users DEFAULT VALUES; Runtime error: NOT NULL constraint failed: users.username (19) sqlite> INSERT INTO users(username) VALUES('james'); Runtime error: NOT NULL constraint failed: users.email (19) sqlite> INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+---------------+ | rowid | username | email | +-------+----------+---------------+ | 1 | james | james@abc.com | +-------+----------+---------------+ sqlite> INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); Runtime error: UNIQUE constraint failed: users.email (19) sqlite> INSERT INTO users(username, email) VALUES('james', 'james_new@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+-------------------+ | rowid | username | email | +-------+----------+-------------------+ | 1 | james | james@abc.com | | 2 | james | james_new@abc.com | +-------+----------+-------------------+ sqlite> INSERT INTO users(username, email) VALUES('jill', 'james_new@abc.com'); Runtime error: UNIQUE constraint failed: users.email (19) sqlite> INSERT INTO users(username, email) VALUES('jill', 'jill@abc.com'); sqlite> SELECT rowid, * FROM users; +-------+----------+-------------------+ | rowid | username | email | +-------+----------+-------------------+ | 1 | james | james@abc.com | | 2 | james | james_new@abc.com | | 3 | jill | jill@abc.com | +-------+----------+-------------------+ sqlite> ``` That is how we primary key works, atleast the basic of it. ## INTEGER PRIMARY KEY This is a special case where the column defined as `PRIMARY KEY` with INTEGER type becomes an alias to the `rowid` and hence we will have both `UNIQUE` and `NOT NULL` constraints added to it. Let's create a table with `INTEGER PRIMARY KEY` column. ```sql CREATE TABLE users ( user_id INTEGER PRIMARY KEY, username TEXT NOT NULL, email TEXT NOT NULL ); ``` In this table `users` we have `user_id` which is `INTEGER PRIMARY KEY` column. It is an alias for the `rowid` table as we'll see. Now, let's insert some data and check what we can observe. ```sql INSERT INTO users DEFAULT VALUES; ``` Obivously, this will fail, as we have not specified the `username` and `email` as `NOT NULL`, and we haven't specified in the insertion. Let's specify both and check if it can violate any constraints. ```sql INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); ``` As you can see, we successfully inserted the record, as we have specified the `username` and `email` as `NOT NULL` and the primary key which is `user_id` and it will automatically get populated. The `user_id` will be automatically populated as it will be a alias to the `rowid` column, and we know it will populate automatically for each inserted value for each row, by incrementing the max row id at that point. ```sql SELECT * FROM users; ``` As you can see, the `user_id` is `1` and `username` is `james` and `email` is `james@abc.com`. But what is `rowid`? Let's check that too. ```sql SELECT rowid, * FROM users; ``` As you can see, the `user_id` is `1` but the rowid has become the `user_id` which is basically an alias for the `rowid`. Let's see if we can add more records. Let's try adding the same record again. ```sql INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); ``` Interestingly, this works, because we have not specified the `user_id` as `PRIMARY KEY` and inserting duplicate username and email should not violate any constraints unless we have specified as `UNIQUE` or others. ``` sqlite> CREATE TABLE users ( user_id INTEGER PRIMARY KEY, username TEXT NOT NULL, email TEXT NOT NULL ); sqlite> INSERT INTO users DEFAULT VALUES; Runtime error: NOT NULL constraint failed: users.username (19) sqlite> INSERT INTO users(username) VALUES('james'); Runtime error: NOT NULL constraint failed: users.email (19) sqlite> INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); sqlite> SELECT * FROM users; +---------+----------+---------------+ | user_id | username | email | +---------+----------+---------------+ | 1 | james | james@abc.com | +---------+----------+---------------+ sqlite> SELECT rowid, * FROM users; +---------+---------+----------+---------------+ | user_id | user_id | username | email | +---------+---------+----------+---------------+ | 1 | 1 | james | james@abc.com | +---------+---------+----------+---------------+ sqlite> INSERT INTO users(username, email) VALUES('james', 'james@abc.com'); sqlite> SELECT rowid, * FROM users; +---------+---------+----------+---------------+ | user_id | user_id | username | email | +---------+---------+----------+---------------+ | 1 | 1 | james | james@abc.com | | 2 | 2 | james | james@abc.com | +---------+---------+----------+---------------+ sqlite> INSERT INTO users(username, email) VALUES('jill', 'jill@abc.com'); sqlite> SELECT rowid, * FROM users; +---------+---------+----------+---------------+ | user_id | user_id | username | email | +---------+---------+----------+---------------+ | 1 | 1 | james | james@abc.com | | 2 | 2 | james | james@abc.com | | 3 | 3 | jill | jill@abc.com | +---------+---------+----------+---------------+ sqlite> ``` So that is how `INTEGER PRIMARY KEY` works. You can add `PRIMARY KEY` to any column, but you need to ensure that it doesn't violate the constraint of allowing duplicate enties. We know that the `PRIMARY KEY` is not a column specific constraint. It is table wide constraint, we'll check out the table wide constraint that is combinational constraint in the next section.

SQLite SQL: CHECK column constraint

Fri, 12 Sep 2025 00:00:00 UTC

## The CHECK column constraint The [CHECK](https://sqlite.org/lang_createtable.html#check_constraints) clause is a column constraint that allows us to define certain conditions that we want to evaluate before inserting the column for the row and populate it accordingly. In this post, we will check, what this check is and when it is performed. ## What it checks The `CHECK` constraint basically acts a validator, we define the condition and if the condition is not met (is false) then the row is not inserted or updated. Let's take a simplest example with our good-ol users table: ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT NOT NULL, age INTEGER CHECK(age >= 18) ); ``` Let's insert a user: ```sql INSERT INTO users (name, age) VALUES ('John', 21); ``` ```sql SELECT * FROM users; ``` This rightly populates with the user `John` with age `21`. If we try to insert a user with age less than 18, it will fail: ```sql INSERT INTO users (name, age) VALUES ('Alice', 17); ``` This should fail the constraint as the age is less than 18. ``` sqlite> INSERT INTO users (name, age) VALUES ('Alice', 17); Runtime error: CHECK constraint failed: age >= 18 (19) sqlite> ``` As you would see, the insertion query failed as a `CHECK` constraint failed. Let's add one more valid user: ```sql INSERT INTO users (name, age) VALUES ('Bob', 20); ``` This is fine, and now, let's try to update the age of a user: ```sql UPDATE users SET age = 17 WHERE id = 2; ``` This should fail the constraint as the age is less than 18. ``` sqlite> INSERT INTO users (name, age) VALUES ('Bob', 20); sqlite> SELECT * FROM users; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | John | 21 | | 2 | Bob | 20 | +----+------+-----+ sqlite> UPDATE users SET age = 17 where id = 2; Runtime error: CHECK constraint failed: age >= 18 (19) sqlite> ``` As you would see, the update query failed as a `CHECK` constraint failed. ## When it is checked? As we saw it is checked before the row is inserted or updated. ## Disabling CHECK constraint We can disable the `CHECK` constraint on all the tables in the database by setting the `PRAGMA` of `ignore_check_constraints` to `TRUE`: ```sql PRAGMA ignore_check_constraints = TRUE; ``` This will disable all the `CHECK` constraints on all the tables in the database. Let's try again updating the user's age with less then 18: ```sql UPDATE users SET age = 17 WHERE id = 2; ``` This time, it readily updates the age to less then 18, without failing the constraint, which means the constraint was disable or shunned down. ``` sqlite> UPDATE users SET age = 17 where id = 2; Runtime error: CHECK constraint failed: age >= 18 (19) sqlite> PRAGMA ignore_check_constraints = TRUE; sqlite> SELECT * FROM users; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | John | 21 | | 2 | Bob | 20 | +----+------+-----+ sqlite> UPDATE users SET age = 17 where id = 2; sqlite> SELECT * FROM users; +----+------+-----+ | id | name | age | +----+------+-----+ | 1 | John | 21 | | 2 | Bob | 17 | +----+------+-----+ sqlite> ``` Similarly the inserts will be allowed if the constraint is disabled. ```sql INSERT INTO users (name, age) VALUES ('Harry', 14); ``` ``` sqlite> INSERT INTO users (name, age) VALUES ('Harry', 14); sqlite> SELECT * FROM users; +----+-------+-----+ | id | name | age | +----+-------+-----+ | 1 | John | 21 | | 2 | Bob | 17 | | 3 | Harry | 14 | +----+-------+-----+ sqlite> ``` ## Table level check constraint The `CHECK` constraint can be applied to a table too, in fact it doesn't really matter you add it to a column or a table, as it will be evaluated for insertion or updation of the column and it's not tied to a column. Let's take a better example: ```sql CREATE TABLE accounts ( id INTEGER PRIMARY KEY, account_type TEXT NOT NULL, balance INTEGER NOT NULL, CHECK ( (account_type = 'savings' AND balance >= 0) OR (account_type = 'loan' AND balance <= 0) ) ); ``` Here, we have a `CHECK` constraint that checks if the account type is `savings` and balance is greater than or equal to 0, or if the account type is `loan` and balance is less than or equal to 0. So we are combining two columns to form a condition and check if the row is actually valid or not to be inserted. Let's insert a few rows: ```sql INSERT INTO accounts (account_type, balance) VALUES ('savings', 1000); INSERT INTO accounts (account_type, balance) VALUES ('loan', -1000); ``` This will insert readily the `savings` and the `loan` account as the constraint passes. However if we try to do the opposite and make the constraint check fail. ```sql INSERT INTO accounts (account_type, balance) VALUES ('savings', -1000); ``` ```sql INSERT INTO accounts (account_type, balance) VALUES ('loan', 1000); ``` It will fail as the check condition is not TRUE anymore for both the cases. ``` sqlite> INSERT INTO accounts (account_type, balance) VALUES ('savings', 1000); sqlite> INSERT INTO accounts (account_type, balance) VALUES ('loan', -1000); sqlite> INSERT INTO accounts (account_type, balance) VALUES ('loan', 1000); Runtime error: CHECK constraint failed: (account_type = 'savings' AND balance >= 0) OR (account_type = 'loan' AND balance <= 0) (19) sqlite> INSERT INTO accounts (account_type, balance) VALUES ('savings', -1000); Runtime error: CHECK constraint failed: (account_type = 'savings' AND balance >= 0) OR (account_type = 'loan' AND balance <= 0) (19) sqlite> ``` As you can see the `CHECK` condition failed for both the cases. ## Check constraint values Note, here the value will be either `TRUE` or `FALSE`. But what about other values? NULL, 0, ? That brings us to the fundamental of how `CHECK` considers the values as `TRUE` and `FALSE`. From the documentation: > If the result is zero (integer value 0 or real value 0.0), then a constraint violation has occurred. > If the CHECK expression evaluates to NULL, or any other non-zero value, it is not a constraint violation So, its not really `TRUE` or `FALSE` but: - `TRUE`: is any non-zero value or `NULL` (if a condition is true, it is set as `1`) - `FALSE`: is `0` (if a condition is false, it is set as `0`) So, let's create a few tables with different `CHECK` constraints having different values for the check constraint: ```sql CREATE TABLE test ( val INTEGER, CHECK(val) ); ``` Now, let's insert some values The below query will evaluate to false, hence failing the constraint. ```sql INSERT INTO test(val) VALUES (0); ``` The value is `1` which is a non-zero value, hence passing the constraint. ```sql INSERT INTO test(val) VALUES (1); ``` Here the value is `0.000` which is still zero, hence failing the constraint. ```sql INSERT INTO test(val) VALUES (0.000); ``` The value here is `0.0001` which is a non-zero value, hence passing the constraint. ```sql INSERT INTO test(val) VALUES (0.0001); ``` The `FALSE` is alias for `0` hence failing the constraint. ```sql INSERT INTO test(val) VALUES (FALSE); ``` Similarly, `TRUE` is an alias for `1` hence passing the constraint. ```sql INSERT INTO test(val) VALUES (TRUE); ``` The empty string is casted as numeric, it evaluates to `0`, hence failing the constraint. ```sql INSERT INTO test(val) VALUES (''); ``` Similarly, the string `'a'` is casted as numeric, it evaluates to `0`, hence failing the constraint. ```sql INSERT INTO test(val) VALUES ('a'); ``` Here's the log, the SQLog :) ```sql sqlite> CREATE TABLE test ( val INTEGER, CHECK(val) ); sqlite> INSERT INTO test(val) VALUES (0); Runtime error: CHECK constraint failed: val (19) sqlite> INSERT INTO test(val) VALUES (1); sqlite> INSERT INTO test(val) VALUES (0.000); Runtime error: CHECK constraint failed: val (19) sqlite> INSERT INTO test(val) VALUES (0.0001); sqlite> INSERT INTO test(val) VALUES (FALSE); Runtime error: CHECK constraint failed: val (19) sqlite> INSERT INTO test(val) VALUES (TRUE); sqlite> INSERT INTO test(val) VALUES (''); Runtime error: CHECK constraint failed: val (19) sqlite> INSERT INTO test(val) VALUES ('a'); Runtime error: CHECK constraint failed: val (19) sqlite> INSERT INTO test(val) VALUES (124); sqlite> SELECT * FROM test; +--------+ | val | +--------+ | 1 | | 0.0001 | | 1 | | 124 | +--------+ sqlite> sqlite> INSERT INTO test(val) VALUES (NULL); sqlite> INSERT INTO test DEFAULT VALUES; sqlite> SELECT * FROM test; +--------+ | val | +--------+ | 1 | | 0.0001 | | 1 | | 124 | | | | | +--------+ sqlite> ``` Now, you might be wondering why the heck `'a'` failed? Well, becuase casting strings in numeric form, will evaluate to `0` ```sql SELECT CAST('a' AS NUMERIC); ``` A wired quirk but worth noting, so won't work with strings, need to cast or perform better checks for string related values. ``` sqlite> INSERT INTO test(val) VALUES ('a'); Runtime error: CHECK constraint failed: val (19) sqlite> SELECT CAST('a' AS NUMERIC); +----------------------+ | CAST('a' AS NUMERIC) | +----------------------+ | 0 | +----------------------+ sqlite> ``` So that is it from the basic walkthrough of the `CHECK` constraint. Some TLDRs: 1. The `CHECK` clause is a column constraint that allows us to define certain conditions that we want to evaluate on the column(s) before inserting/updating the row. 2. The constraint is table wide, as it boils down to inserting/updating or not inserting/updating the row. 3. The `CHECK` constraint is evaluated before the row is inserted/updated. 4. The value of the conditions is evaluated as a `NUMERIC` value so any value is `TRUE` if it is not `0`, even `NULL` is true, and `FALSE` if it is `0`. 5. The `CHECK` constraint can be disabled using the `PRAGMA` command `PRAGMA ignore_check_constraints = TRUE;` or `PRAGMA ignore_check_constraints = ON;` `ON`, `TRUE`, either works.

SQLite SQL: GENERATED column constraint

Thu, 11 Sep 2025 00:00:00 UTC

## The GENERATED column constraint We saw certain limitations of the `DEFAULT` column constraint clause, that it can't generate dynamic values based on other columns or some other dynamic popularities. For overcomming those kind of constraints, there is one more specific constraint, the [GENERATED](https://sqlite.org/gencol.html) column constraint. You can think of it as a `DEFAULT` but having more than static values, it is evaluated and computed each time a new row is inserted. However they come with one catch, you can't later update the values, you need to update the values of the dependent columns first, so it will generate **ALWAYS** a update is made to the row(relevant columns in that row). Let's create a simple table to understand the `GENERATED` column constraint: ```sql CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT, word_count INTEGER GENERATED ALWAYS AS (length(content) - length(replace(content, ' ', '')) + 1), read_time_minutes INTEGER GENERATED ALWAYS AS (ceil(word_count / 200.0)) ); ``` The syntax of the `GENERATED` constraint is ``` column_name GENERATED ALWAYS AS ``` Nothing fancy, just a content field as a text, then based on that value, the `word_count` and `read_time_minutes` is always calculated/generated based on the value of the `content` and the `read_time_minutes` is inturn dependent on the `word_count` value. So, the change in `content` will make a update on the `word_count` that will inturn update the `read_time_minutes`, like a chain reaction of sorts. Let's insert a sample post, note, we can't insert a generated value. ```sql INSERT INTO posts(content) VALUES('A sample post'); ``` This will create a record with the `id` as 1, the `content` as `A sample post`, and the `word_count` will be generated/calculated as the `(length(content) - length(replace(content, ' ', ''))+1)` as the number of spaces + 1. The `read_time_minutes` is generated with the `word_count` divided by 200 which is the average word read in a minute by a human (let's not go into the nity-grity) but you can get the point of this. These are just mathematical values, and those are dependent on the value of the other columns of the same record/row. ```sql SELECT * FROM posts; ``` As, you can see here, the value of `word_count` is `3` and the value of `read_time_minutes` is `1`. Let's insert a few more values and check how these generated values behave. Let's insert a bit longer text. ```sql INSERT INTO posts(content) VALUES('A SQLITE Post for the generated column constraint.'); ``` ```sql SELECT * FROM posts; ``` As, you can see the value of the `word_count` is `8` and the value of `read_time_minutes` is `1`. Let's update the `content` and check if the `word_count` and the `read_time_minutes` are updated. ```sql UPDATE posts SET content = content || 'New content some more content' where id = 2; SELECT * FROM posts; ``` As you can see the value of the `word_count` is `11` and the value of `read_time_minutes` is `1` still. That is because, until we write `200` words it will ceil off to 1. Let's add a record with more than `200` words, and check if the `read_time_minutes` is updated. ```sql INSERT INTO posts(content) VALUES('New post'); SELECT * FROM posts; ``` The current value of the `word_count` is `2` and the `read_time_minutes` is `1`, let's update it by adding the value of `Adding some extra words here.`. ```sql UPDATE posts SET content = content || replace( printf('%200c', ' '), ' ', ' Adding some extra words here.' ) WHERE id = 3; ``` Ok, this might be too much, but let's break it down: - First, we are updating the `content` by adding the value of `Adding some extra words here.` to the end of the existing value of `content`. - In the replace function, the parameters are original string, the value to replace, and the value to replace it with. - So in this case, we are taking the string `printf('%200c', ' ')`, which is a string of 200 spaces, basically adding a single space character, but it will be right-justified within a field of 200 characters. - The second parameter is the character to replace, in this case it is single white space or `' '` character. - The third parameter is the text to replace that `' '` space character with, here it is ` Adding some extra words here`. - So, essentially we are taking `200` white spaces and adding ` Adding some extra words here.` for each space to the end of the existing value of `content`. Hence we will have around ~`200 * 5` words i.e. 1000 words, this way we can check the value of the `read_time_minutes` is updated. ```sql SELECT * FROM posts; ``` As you can see the value of the `word_count` is `1002` and the value of `read_time_minutes` is `6`. That is perfect. And it is getting updated for each updation to the `content` column. Here's a step by step output log: Its a SQLog :) ``` sqlite> CREATE TABLE posts ( id INTEGER PRIMARY KEY, content TEXT, word_count INTEGER GENERATED ALWAYS AS (length(content) - length(replace(content, ' ', '')) + 1), read_time_minutes INTEGER GENERATED ALWAYS AS (ceil(word_count / 200.0)) ); sqlite> .mode table sqlite> INSERT INTO posts(content) VALUES('A sample post'); sqlite> SELECT * FROM posts; +----+---------------+------------+-------------------+ | id | content | word_count | read_time_minutes | +----+---------------+------------+-------------------+ | 1 | A sample post | 3 | 1 | +----+---------------+------------+-------------------+ sqlite> INSERT INTO posts(content) VALUES('A SQLITE Post for the generated column constraint.'); sqlite> SELECT * FROM posts; +----+----------------------------------------------------+------------+-------------------+ | id | content | word_count | read_time_minutes | +----+----------------------------------------------------+------------+-------------------+ | 1 | A sample post | 3 | 1 | | 2 | A SQLITE Post for the generated column constraint. | 8 | 1 | +----+----------------------------------------------------+------------+-------------------+ sqlite> UPDATE posts SET content = content || 'New content ' ...> some more content' where id = 2; sqlite> SELECT * FROM posts; +----+--------------------------------------------------------------+------------+-------------------+ | id | content | word_count | read_time_minutes | +----+--------------------------------------------------------------+------------+-------------------+ | 1 | A sample post | 3 | 1 | +----+--------------------------------------------------------------+------------+-------------------+ | 2 | A SQLITE Post for the generated column constraint.New conten | 11 | 1 | | | t | | | | | some more content | | | +----+--------------------------------------------------------------+------------+-------------------+ sqlite> INSERT INTO posts(content) VALUES('New post'); sqlite> SELECT * FROM posts; +----+--------------------------------------------------------------+------------+-------------------+ | id | content | word_count | read_time_minutes | +----+--------------------------------------------------------------+------------+-------------------+ | 1 | A sample post | 3 | 1 | +----+--------------------------------------------------------------+------------+-------------------+ | 2 | A SQLITE Post for the generated column constraint.New conten | 11 | 1 | | | t | | | | | some more content | | | +----+--------------------------------------------------------------+------------+-------------------+ | 3 | New post | 2 | 1 | +----+--------------------------------------------------------------+------------+-------------------+ sqlite> UPDATE posts SET content = content || replace(printf('%200c', ' '), ' ', ' Adding some extra words here.') WHERE id = 3; sqlite> SELECT * FROM posts; +----+--------------------------------------------------------------+------------+-------------------+ | id | content | word_count | read_time_minutes | +----+--------------------------------------------------------------+------------+-------------------+ | 1 | A sample post | 3 | 1 | +----+--------------------------------------------------------------+------------+-------------------+ | 2 | A SQLITE Post for the generated column constraint.New conten | 11 | 1 | | | t | | | | | some more content | | | +----+--------------------------------------------------------------+------------+-------------------+ | 3 | New post Adding some extra words here. Adding some extra wor | 1002 | 6 | | | ds here. Adding some extra words here. Adding some extra wor | | | | | ds here. Adding some extra words here. Adding some extra wor | | | | |..............................................................| | | | | ds here. Adding some extra words here. Adding some extra wor | | | | | ds here. | | | +----+--------------------------------------------------------------+------------+-------------------+ ``` ## Generated columns cannot be updated manually If you tried to update the columns with `GENERATED ALWAYS` constraint, you will get a `constraint failed` error. As the constraint itself says, it is **GENERATED ALWAYS**, hence not to be updated or inserted. ```sql UPDATE posts SET word_count = 10 WHERE id = 1; ``` It will fail with `cannot UPDATE generated column "word_count"` as mentioned. ```sql UPDATE posts SET read_time_minutes = 10 WHERE id = 1; ``` Similarly, this will also fail, as it is also a `GENERATED ALWAYS` constrained column. ``` sqlite> UPDATE posts SET word_count = 10 WHERE id = 1; Parse error: cannot UPDATE generated column "word_count" sqlite> UPDATE posts SET read_time_minutes = 10 WHERE id = 1; Parse error: cannot UPDATE generated column "read_time_minutes" ``` ## STORED VS VIRTUAL Now, the Generated columns can be `STORED` as in stored in the database to the disk file, or they could be `VIRTUAL` as in computed (generated,calculated each time). There are trade-offs and I will definitely add the meme here `IT ACTUALLY DEPENDS`! The `STORED` column will take up space but the queries will be quick, since it doesn't need to calculate it each time, only it needs to do it at each updation (that can't be avoided). The `VIRTUAL` column will not take up space but the queries will be a little slower, depending on the number of data to compute, since it doesn't store the column values, it will have to compute each time and it as well need to update it each time. > The default is `VIRTUAL` if not specified. The commonality of `STORED` and `VIRTUAL` is that they are both updated at the updation of the column it relates to or the updation of the row, the only difference is that one stores it the other calculates it for each query made to it. We'll see the difference how is that the case, it generates each time in the case of the `VIRTUAL` column `GENERATED` constraint. I'll show the most basic example continuing with the same table, but with the `VIRTUAL` clause on the `GENERATED ALWAYS` constraint. ### Virtual generated column By default, it is `VIRTUAL` it doesn't matter if you add it or not (let's add it to make it clear). I'll specify the `word_count` as the `VIRTUAL` column with the `GENERATED ALWAYS` constraint with the same formula to get the word count and at the end specify the `VIRTUAL` keyword (its optional). > Let's skip the `read_time_minutes` for now, you can add it won't make it any difference (it was a little extreme example to use 200 words to update its value) ```sql CREATE TABLE posts_virtual ( id INTEGER PRIMARY KEY, content TEXT, word_count INTEGER GENERATED ALWAYS AS ( length(content) - length(replace(content, ' ', '')) + 1 ) VIRTUAL ); ``` Now, we'll add some data to the table and check the value of the `word_count` column. ```sql INSERT INTO posts_virtual(content) VALUES('A sample post'); INSERT INTO posts_virtual(content) VALUES('A SQLITE Post for the virtual generated column constraint.'); SELECT * FROM posts_virtual; ``` This we already know, the plain old `word_count` is `3` and `9` for the two posts respectively. But let's see how is the `word_count` column is actually stored and how slow or fast it actually is. ```sql PRAGMA table_info(posts_virtual); ``` HH? There is no `word_count` column here, it is not stored, it is a `VIRTUAL` column. ``` sqlite> PRAGMA table_info(posts_virtual); +-----+---------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+---------+---------+---------+------------+----+ | 0 | id | INTEGER | 0 | | 1 | | 1 | content | TEXT | 0 | | 0 | +-----+---------+---------+---------+------------+----+ sqlite> ``` What about extra info? ```sql PRAGMA table_xinfo(posts_virtual); ``` Yes, that is indeed there, and its hidden? Nope, its `generated` a special type. - `0` is not hidden - `1` is hidden - `2` is generated ``` sqlite> PRAGMA table_xinfo(posts_virtual); +-----+------------+---------+---------+------------+----+--------+ | cid | name | type | notnull | dflt_value | pk | hidden | +-----+------------+---------+---------+------------+----+--------+ | 0 | id | INTEGER | 0 | | 1 | 0 | | 1 | content | TEXT | 0 | | 0 | 0 | | 2 | word_count | INTEGER | 0 | | 0 | 2 | +-----+------------+---------+---------+------------+----+--------+ sqlite> ``` It's time to benchmark it then, let's bulk insert 100000 posts. ```sql WITH RECURSIVE cnt(x) AS ( SELECT 1 UNION ALL SELECT x+1 FROM cnt WHERE x < 100000 ) INSERT INTO posts_virtual(content) SELECT 'This is a sample post number ' || x || ' with some words repeated multiple times.' FROM cnt; ``` If we see now the `posts_virtual` table has over 100000 rows. ```sql SELECT COUNT(*) FROM posts_virtual; ``` We already had `2` records, and we added `100000` more, so the total is `100002`. Let's check the sum of the `word_count` to be sure it inserted it right. ```sql SELECT SUM(word_count) FROM posts_virtual; ``` Ok that shows the sum of the `word_count` is `1300012`. Let's check with the timer on and evaluate the query. ``` sqlite> WITH RECURSIVE cnt(x) AS ( SELECT 1 UNION ALL SELECT x+1 FROM cnt WHERE x < 100000 ) INSERT INTO posts_virtual(content) SELECT 'This is a sample post number ' || x || ' with some words repeated multiple times.' FROM cnt; sqlite> PRAGMA table_xinfo(posts_virtual); +-----+------------+---------+---------+------------+----+--------+ | cid | name | type | notnull | dflt_value | pk | hidden | +-----+------------+---------+---------+------------+----+--------+ | 0 | id | INTEGER | 0 | | 1 | 0 | | 1 | content | TEXT | 0 | | 0 | 0 | | 2 | word_count | INTEGER | 0 | | 0 | 2 | +-----+------------+---------+---------+------------+----+--------+ sqlite> .timer on sqlite> SELECT COUNT(*) FROM posts_virtual; +----------+ | COUNT(*) | +----------+ | 100002 | +----------+ Run Time: real 0.001 user 0.000352 sys 0.000103 sqlite> SELECT SUM(word_count) FROM posts_virtual; +-----------------+ | SUM(word_count) | +-----------------+ | 1300012 | +-----------------+ Run Time: real 0.041 user 0.040682 sys 0.000418 sqlite> PRAGMA page_count; +------------+ | page_count | +------------+ | 2086 | +------------+ Run Time: real 0.000 user 0.000000 sys 0.000078 sqlite> PRAGMA page_size; +-----------+ | page_size | +-----------+ | 4096 | +-----------+ Run Time: real 0.000 user 0.000025 sys 0.000006 sqlite> UPDATE posts_virtual SET content = content || ' extra text' WHERE id % 1000 = 0; Run Time: real 0.010 user 0.009353 sys 0.000161 ``` I think we'll leave it here and move onto creating the `STORED` column as a generated value ### Stored generated column Similar to the `VIRTUAL` column, we'll create the `STORED` column with the `GENERATED ALWAYS` constraint. This will be the same table named as `posts_stored` but with the `word_count` being a `STORED` column rather than a `VIRTUAL` column. ```sql CREATE TABLE posts_stored ( id INTEGER PRIMARY KEY, content TEXT, word_count INTEGER GENERATED ALWAYS AS ( length(content) - length(replace(content, ' ', '')) + 1 ) STORED ); ``` Now, we'll add some data to the table. ```sql INSERT INTO posts_stored(content) VALUES('A sample post'); INSERT INTO posts_stored(content) VALUES('A SQLITE Post for the virtual generated column constraint.'); SELECT * FROM posts_stored; ``` This is cool, working as expected, we have already seen it, we need to take a look at how this values are stored now. Let's take a look at the `table_info` for the `posts_stored` table. ```sql PRAGMA table_info(posts_stored); ``` It still doesn't have the `GENERATED` column `word_count`. Let's take a look at the `table_xinfo` for the `posts_stored` table. ```sql PRAGMA table_xinfo(posts_stored); ``` Ok, wiredly it has the `GENERATED ALWAYS` column `word_count` as well but the `hidden` value is `3`. We knew it was `2` for hidden but `3` is for? You guessed it right, stored and generated column. Now, let's also populate the `posts_stored` table with some data, bulk inserting the data. Let's add `100000` rows in it. ```sql WITH RECURSIVE cnt(x) AS ( SELECT 1 UNION ALL SELECT x+1 FROM cnt WHERE x < 100000 ) INSERT INTO posts_stored(content) SELECT 'This is a sample post number ' || x || ' with some words repeated multiple times.' FROM cnt; ``` This is the same query except its inserting in the `posts_stored` table. Let's check if it has inserted the right number of rows. ```sql SELECT COUNT(*) FROM posts_stored; ``` Indeed there are `100000` rows in the `posts_stored` table. Now, let's verify that the inserted values are rightly populated. ```sql SELECT SUM(word_count) FROM posts_stored; ``` This gives `1300012` as the sum of the `word_count` from the `posts_stored` table. Which matches the sum of the `word_count` from the `posts_virtual` table. ```sql sqlite> CREATE TABLE posts_stored ( id INTEGER PRIMARY KEY, content TEXT, word_count INTEGER GENERATED ALWAYS AS ( length(content) - length(replace(content, ' ', '')) + 1 ) STORED ); Run Time: real 0.000 user 0.000258 sys 0.000027 sqlite> INSERT INTO posts_stored(content) VALUES('A sample post'); sqlite> INSERT INTO posts_stored(content) VALUES('A SQLITE Post for the virtual generated column constraint.'); SELECT * FROM posts_stored; +----+------------------------------------------------------------+------------+ | id | content | word_count | +----+------------------------------------------------------------+------------+ | 1 | A sample post | 3 | | 2 | A SQLITE Post for the virtual generated column constraint. | 9 | +----+------------------------------------------------------------+------------+ Run Time: real 0.000 user 0.000224 sys 0.000000 sqlite> PRAGMA table_info(posts_stored); +-----+---------+---------+---------+------------+----+ | cid | name | type | notnull | dflt_value | pk | +-----+---------+---------+---------+------------+----+ | 0 | id | INTEGER | 0 | | 1 | | 1 | content | TEXT | 0 | | 0 | +-----+---------+---------+---------+------------+----+ sqlite> PRAGMA table_xinfo(posts_stored); +-----+------------+---------+---------+------------+----+--------+ | cid | name | type | notnull | dflt_value | pk | hidden | +-----+------------+---------+---------+------------+----+--------+ | 0 | id | INTEGER | 0 | | 1 | 0 | | 1 | content | TEXT | 0 | | 0 | 0 | | 2 | word_count | INTEGER | 0 | | 0 | 3 | +-----+------------+---------+---------+------------+----+--------+ Run Time: real 0.000 user 0.000013 sys 0.000322 sqlite> SELECT COUNT(*) FROM posts_stored; +----------+ | COUNT(*) | +----------+ | 2 | +----------+ Run Time: real 0.000 user 0.000108 sys 0.000012 sqlite> WITH RECURSIVE cnt(x) AS ( SELECT 1 UNION ALL SELECT x+1 FROM cnt WHERE x < 100000 ) INSERT INTO posts_stored(content) SELECT 'This is a sample post number ' || x || ' with some words repeated multiple times.' FROM cnt; Run Time: real 0.071 user 0.067858 sys 0.002929 sqlite> SELECT COUNT(*) FROM posts_stored; +----------+ | COUNT(*) | +----------+ | 100002 | +----------+ Run Time: real 0.000 user 0.000367 sys 0.000000 sqlite> SELECT SUM(word_count) FROM posts_stored; +-----------------+ | SUM(word_count) | +-----------------+ | 1300012 | +-----------------+ Run Time: real 0.008 user 0.006492 sys 0.000941 ``` This is all setup for the comparison of both the tables. ### The difference Now, let's check the difference between the `STORED` and `VIRTUAL` column. Let's sum the `word_count` from each of the tables `posts_stored` and `posts_virtual` First, lets set the timer on, which will give the breakdown of the time for each ran query. ``` .timer on ``` The timer will give three time durations, the real, user and the system. The breakdown is like this: - real time : The full clock time form the start to the end of the query - user time : The CPU time in the user space (I/O) operations on the sqlite side, computations on the obtained data, or preprocessing the data before inserting. - system time: The CPU time in the kernel space (I/O) operations on the host side, it could be reading or writing to the actual db file, etc. And then run the query for the `posts_stored` table. ```sql SELECT SUM(word_count) FROM posts_stored; ``` This is giving the result in around `~0.008` seconds might be approximately `0.01` in certain cases. We can even get the breakdown of the time for each ran query. - full query execution time: `0.008` - user time: `0.006492` - system time: `0.000941` So, here in this case, the system and user time are around the same, however there is some system time for reading from the database file. And then run the query for the `posts_virtual` table. ```sql SELECT SUM(word_count) FROM posts_virtual; ``` This is giving the result in around `~0.039` seconds might be approximately `0.04` in certain cases. We can even get the breakdown of the time for each ran query. - full query execution time: `0.039` - user time: `0.039366` - system time: `0.000000` this can go sometimes to `0.000652`, almost negligible, but if it happens, in that case the full query time will increase by a bit. So, it is taking a quite lot of time in the user space by the CPU right? And almost 0 time in the kernel level (disk operation) to read from the database file. That is so clear, as this is a virtual column, and we are only reading a single column, and that too is a virtual column, we don't need to hit a read to the database file. Clever right? But that costs some CPU to compute the sum for the `100002` rows. This is the trade-off - Time spent on the CPU on the computation vs the time spent on the disk operation to read the data. - These can be different depending on the query. ```sql sqlite> SELECT SUM(word_count) FROM posts_stored; +-----------------+ | SUM(word_count) | +-----------------+ | 1300012 | +-----------------+ Run Time: real 0.008 user 0.006492 sys 0.000941 sqlite> SELECT SUM(word_count) FROM posts_virtual; +-----------------+ | SUM(word_count) | +-----------------+ | 1300012 | +-----------------+ Run Time: real 0.039 user 0.039366 sys 0.000000 ``` You can clearly see the difference, the `STORED` column is almost `4` to `5` times faster than the `VIRTUAL` column. ### Key observations 1. If you have lot of processing to do in the generated column, better to go with the `STORED` column. 2. If you have a lot of data to compute and not necessarily complex, better to go with the `VIRTUAL` column. 3. It really depends on the data too, and we cannot say which one is better over the other without knowing the structure and the design of the queries. There are lot of details as well beyond this, it will require specific guides on how to do certain things and busting certain myths in the documentation.

SQLite SQL: DEFAULT column constraint

Wed, 10 Sep 2025 00:00:00 UTC

## DEFAULT Constraint The `DEFAULT` constraint in SQLite is used to specify a default value for a column, if not provided, then the column will be populated with the `NULL` value as the default value. According to the docs: > The DEFAULT clause specifies a default value to use for the column if no value is explicitly provided by the user when doing an INSERT. If there is no explicit DEFAULT clause attached to a column definition, then the default value of the column is NULL. NOTE: This constraint doesn't guard a `NULL` value, it only prevents the initial insert (or new record creation) of a `NULL` value, you can update the column later with a `NULL` value. It only is used to override the default `NULL` value, by providing a different one. The `DEFAULT` constraint can only take constant values, and it can't deduce or dynamically store values at the time of creation, if you want that kind of default values, then the other constraint [GENERATED ALWAYS](https://sqlite.org/lang_createtable.html#the_generated_always_as_clause) would help. > Also, DEFAULT might not be a constraint, its just a clause or a column constraint clause. ## Creating a Table with a DEFAULT Constraint The general syntax for creating a default constraint is basically the `DEFAULT` keyword, followed by the actual literal value. In the below example, let's create a dummy users table, with the `country_code` and `followers` as the columns, and their default values as `US` and `0` respectively. Note that the value can be wrapped in a `()` but it's not a expression so we can avoid it. However, writing the value in the `()` makes it more redable I think. ```sql CREATE TABLE users ( username TEXT NOT NULL UNIQUE, country_code TEXT DEFAULT 'US', followers INTEGER DEFAULT 0 ); ``` Once we have a table with columns having default values, we can avoid inserting the values each time, as the default values will be populated. ```sql INSERT INTO users(username) VALUES('alice'); ``` Now, we have inserted a row with the `username` as `alice`, and the `country_code` and `followers` will be populated as the default values as they are not specified as `US` and `0` respectively. ```sql SELECT * FROM users; ``` It doens't mean that you can't add any other values, the default values will be used when you don't provide any value while inserting the row. In other words, the problem of seeing empty values (`NULL` values) when you don't specify will be gone, unless you update the record later explicitly with the `NULL` or empty values. ```sql INSERT INTO users(username, country_code) VALUES('bob', 'UK'); INSERT INTO users(username, country_code, followers) VALUES('ronaldo', 'PT', 100000); ``` In the above queries, we can set the `country_code` or the `followers` if we want to insert a specific value in the `country_code` or the `followers` column. ```sql SELECT * FROM users; ``` As you can see, it populates the values if provided, else uses the default values. ## Inserting default values If you want to create a record with all default values, you can specify the `DEFAULT VALUES` as the statement clause in the `INSERT` statement. For this to work, we need to have a default value for every column. If not specified it will be `NULL`, but if there are columns with `NOT NULL` or other constraint that needs to have some value from the user then we won't be able to insert the record without providing at least one value. ```sql INSERT INTO users DEFAULT VALUES; ``` This above statement will fail because we don't have any default value for the `username` column and it has a `NOT NULL` constraint, as discussed in the [NOT NULL Constraint blog post](https://www.meetgor.com/sqlog/sqlite-not-null-column-constraint/), we need to provide a value for that column in order to create a new record. So, it will only we suited, for tables having all values with some or the other values default or generated. Like the example below: ```sql CREATE TABLE documents ( id INTEGER PRIMARY KEY, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, title TEXT DEFAULT 'Untitled', content TEXT DEFAULT 'Here goes the content' ); ``` In the above table, all the columns have one or the other `default` value, like `PRIMARY KEY` this is a alias for the underlying `rowid` column (if not mentioned explicitly or with combinational columns). The default value for `created_at` at is `CURRENT_TIMESTAMP` which will look in a second, the `title` and `content` has default value of `Untitled` and `Here foes the content` respectively. ```sql INSERT INTO documents DEFAULT VALUES; ``` Now, we can insert the `DEFAULT VALUES` without any value specified in the `VALUES` or columns mentioned in the statement. ```sql SELECT * FROM documents; ``` As you can see, now we can insert multiple rows with default values, and the unique id will keep the records in the `documents` table unique. ## Defaults for Date and Time related columns There are special values or constant expression that can be used as default expressions like - `CURRENT_TIME` value in the format `HH:MM:SS` - `CURRENT_DATE` value in the format `YYYY-MM-DD` - `CURRENT_TIMESTAMP` value in the format `YYYY-MM-DD HH:MM:SS` These values can be used as default values relating to date and time related columns which are mostly text related fields. ```sql CREATE TABLE documents ( id INTEGER PRIMARY KEY, created_at DATETIME DEFAULT CURRENT_TIMESTAMP, updated_at DATETIME DEFAULT CURRENT_TIMESTAMP, title TEXT DEFAULT 'Untitled', content TEXT DEFAULT 'Here goes the content' ); ``` The various values can be used here, like for instance, `created_at`, `updated_at` which can either be `CURRENT_TIMESTAMP` or just `CURRENT_DATE` depending on the use case. ```sql INSERT INTO documents DEFAULT VALUES; ``` Here, it will insert a record with the `created_at` and `updated_at` as the current date and time values. ```sql SELECT * FROM documents; ``` ``` sqlite> SELECT * FROM documents; +----+---------------------+---------------------+----------+-----------------------+ | id | created_at | updated_at | title | content | +----+---------------------+---------------------+----------+-----------------------+ | 1 | 2025-09-10 17:04:43 | 2025-09-10 17:04:43 | Untitled | Here goes the content | +----+---------------------+---------------------+----------+-----------------------+ sqlite> ``` As you can see, now we can insert multiple rows with default values, and the unique id will keep the records in the `documents` table unique. That's the basic we can use the `DEFAULT` constraint, you can add any expression as long as it produces a constant value for each of the records.

SQLite SQL: UNIQUE column constraint

Tue, 09 Sep 2025 00:00:00 UTC

## Unique Constraint The `UNIQUE` constraint ensures that a column in a table contains no duplicate values. Like, if you add an `UNIQUE` constraint to a column, then no two rows can have the exact/same value for that column. The documentation is more nuanced. > A `UNIQUE` constraint is similar to a `PRIMARY KEY` constraint, **except** that a single table may have any number of `UNIQUE` constraints. For each `UNIQUE` constraint on the table, each row must contain a unique combination of values in the columns identified by the `UNIQUE` constraint. For the purposes of `UNIQUE` constraints, `NULL` values are considered distinct from all other values, including other NULLs. As with `PRIMARY KEY`s, a `UNIQUE` table-constraint clause must contain only column names — the use of expressions in an indexed-column of a `UNIQUE` table-constraint is not supported. There are three points to note, I'll simplify the points: 1. If you add an `UNIQUE` constraint to a column, then no two rows can have the exact/same value for that column in that table. 2. The `UNIQUE` constraint considers `NULL` values as distinct i.e. it cannot distinguish between two rows having `NULL` values, hence two rows will pass the `UNIQUE` constraint. (Better to use `UNIQUE` and `NOT NULL`) 3. The `UNIQUE` constraint is tied to one or more columns, so you cannot add custom expressions to the `UNIQUE` constraint. (Better to use `UNIQUE INDEX` to create custom expression-based uniqueness or index for querying data.) ## Adding a UNIQUE constraint To add a unique constraint to a column, simply add the `UNIQUE` keyword to the column definition: Let's keep the table simple with just three fields: `name`, `email`, and `age`. ```sql CREATE TABLE users ( name TEXT, email TEXT UNIQUE, age INTEGER CHECK (age > 0) ); ``` Now, lets insert a few records ```sql INSERT INTO users (name, email, age) VALUES('alice', 'alice@wonderland.com', 12); INSERT INTO users (name, email, age) VALUES('bob', 'bob@wonderland.com', 13); ``` Now, lets try to insert a record with the existing email value: ```sql INSERT INTO users (name, email, age) VALUES('charlie', 'alice@wonderland.com', 14); ``` This will immediately fail the `UNIQUE` constraint on the email, and the record will not be inserted. ``` sqlite> INSERT INTO users (name, email, age) VALUES('charlie', 'alice@wonderland.com', 14); Runtime error: UNIQUE constraint failed: users.email (19) ``` Notice, as the constraint is only on the email column, it's not on the name or age columns. ```sql INSERT INTO users (name, email, age) VALUES('alice', 'charlie@wonderland.com', 12); ``` This will succeed, as the `UNIQUE` constraint is only on the email column. Hence, we can conclude that the `UNIQUE` constraint is on a column level when stated next to the column while creating the table. ```sql SELECT * FROM users; ``` ``` +-------+------------------------+-----+ | name | email | age | +-------+------------------------+-----+ | alice | alice@wonderland.com | 12 | | bob | bob@wonderland.com | 13 | | alice | charlie@wonderland.com | 12 | +-------+------------------------+-----+ sqlite> ``` Let's understand the problem first with `NULL` values in UNIQUE constraint. ## Adding a UNIQUE constraint with NULL values What if I insert a record with a `NULL` value for the `email` column? ```sql INSERT INTO users (name, email, age) VALUES('charlie', NULL, 14); --OR INSERT INTO users (name, age) VALUES('charlie', 14); ``` That went in, it created the record, which is fine, so far, as there was no previous entry with `NULL` email in the table, or rather the `email` column. ```sql SELECT * FROM users; ``` However, what happens if we try to insert another record with a `NULL` value for the `email` column? ```sql INSERT INTO users (name, email, age) VALUES('dave', NULL, 15); -- OR INSERT INTO users (name, age) VALUES('dave', 15); ``` This goes in as well, Hhh? It shouldn't have right? If `alice@wonderland.com` is not allowed twice then why should `NULL` be allowed? That's what the documentation clearly stated: > For the purposes of `UNIQUE` constraints, `NULL` values are considered distinct from all other values, including other NULLs. That is clear as crystal, SQLite won't distinguish between two `NULL` values in the `email` column. Hence, we can't have duplicate `NULL` values in the `email` column. For that we need to set email as `NOT NULL` as well as `UNIQUE` inorder to avoid populating `NULL` values in the `email` column. ## Adding UNIQUE Constraint on multiple columns If you wondered, can we add unique constraint on multiple columns? Yes, we can and there are subtle variations on how you interpret those as, multiple as in two or more columns unique independently or the combination of two or more columns being unique. 1. Unique constraint independent of each other 2. Combinational Unique constraint ### Unique constraint independent of each other Let's say we want to add constraint to the user table, which could have more than one column that needs to be unique, like example, the `phone_number` for two users cannot be same, and so on. ```sql DROP TABLE users; CREATE TABLE users ( name TEXT, email TEXT UNIQUE, phone_number TEXT UNIQUE, government_id TEXT UNIQUE, age INTEGER CHECK (age > 0), bio TEXT ); ``` In the above table schema, we have three unique constraints, `email`, `phone_number`, and `government_id`. Each unique constraint is independent of each other. Let's take a look at a few cases in order to understand it better. ```sql INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('bob', 'bob@wonderland.com', '1234567891', 'DEF456', 13, 'I am bob'); ``` ```sql SELECT * FROM users; ``` That is the simple case, However what would happen if one user has the same `phone_number`, and `government_id` but different `email`? ```sql INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'new_alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); ``` ``` sqlite> INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'new_alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); Runtime error: UNIQUE constraint failed: users.government_id (19) ``` Nope, it is independent of each constraint, no two rows can have the same `government_id` or `phone_number` and even the `email` column. Now, what if we want to allow `phone_number`, `government_id` and `email_id` to have a unique combination? Like the three of the columns together should be unique across the table, now with that they won't be unique across the single column, they are combined. ### Combinational Unique constraint In a combinational unique constraint, we can add two or more columns like `UNIQUE(colum_1, column_2, ... column_n)` as a combinational constraint. In this type of constraint: - Not necessarily each column needs to be unique, the combination of the `N` columns should be unique ```sql DROP TABLE users; CREATE TABLE users ( name TEXT, email TEXT NOT NULL, phone_number TEXT NOT NULL, government_id TEXT NOT NULL, age INTEGER CHECK (age > 0), bio TEXT, UNIQUE(email, phone_number, government_id) ); -- OR -- CREATE TABLE users( -- name TEXT, -- email TEXT NOT NULL, -- phone_number TEXT NOT NULL, -- age INTEGER CHECK (age > 0), -- bio TEXT, -- UNIQUE(email, phone_number) -- ); ``` I am just making it up, its not secure, like combination of `email`, `phone_number` and `government_id` is not secure, you can debate about that, but it's not a data-integrity and schema-design class, this post is about understanding the `UNIQUE` column constraint with combination of two or more columns. ```sql INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('bob', 'bob@wonderland.com', '1234567891', 'DEF456', 13, 'I am bob'); INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'new_alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); ``` ```sql SELECT * FROM users; ``` That works! We have `1234567890` as the `phone_number` and `ABC123` as the `government_id` which is same, but `email` is different. So, the combination of `email`, `phone_number` and `government_id` is unique. ``` sqlite> SELECT * FROM users; +-------+--------------------------+--------------+---------------+-----+------------+ | name | email | phone_number | government_id | age | bio | +-------+--------------------------+--------------+---------------+-----+------------+ | alice | alice@wonderland.com | 1234567890 | ABC123 | 12 | I am alice | | bob | bob@wonderland.com | 1234567891 | DEF456 | 13 | I am bob | | alice | new_alice@wonderland.com | 1234567890 | ABC123 | 12 | I am alice | +-------+--------------------------+--------------+---------------+-----+------------+ sqlite> INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('bob', 'new_alice@wonderland.com', '0987654321', 'ABC123', 12, 'I am alice'); sqlite> SELECT * FROM users; +-------+--------------------------+--------------+---------------+-----+------------+ | name | email | phone_number | government_id | age | bio | +-------+--------------------------+--------------+---------------+-----+------------+ | alice | alice@wonderland.com | 1234567890 | ABC123 | 12 | I am alice | | bob | bob@wonderland.com | 1234567891 | DEF456 | 13 | I am bob | | alice | new_alice@wonderland.com | 1234567890 | ABC123 | 12 | I am alice | | bob | new_alice@wonderland.com | 0987654321 | ABC123 | 12 | I am alice | +-------+--------------------------+--------------+---------------+-----+------------+ sqlite> ``` ```sql INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('alice', 'new_alice@wonderland.com', '1234567890', 'ABC123', 12, 'I am alice'); ``` ``` sqlite> INSERT INTO users (name, email, phone_number, government_id, age, bio) VALUES('bob', 'new_alice@wonderland.com', '0987654321', 'ABC123', 12, 'I am alice'); Runtime error: UNIQUE constraint failed: users.email, users.phone_number, users.government_id (19) sqlite> ``` Now this will fail, as the `email`, `phone_number`, `government_id` combination is already unique. ## Primary key vs Unique Constraint Both of these do the same thing, but `UNIQUE` can be added to multiple columns however, the `PRIMARY KEY` should only be one, but it can contain multiple columns. In other words, UNIQUE constraint is column-level, however each table can only have one `PRIMARY KEY` that will be used to indentify the uniqueness among the rows. ``` PRIMARY KEY = UNIQUE + NOT NULL ``` Primary key is also a shorthand for `UNIQUE` and `NOT NULL`, if you add a `PRIMARY KEY` to a column or combination of columns for a table, you needn't specify `UNIQUE` and `NOT NULL`, it already does that for you. So, that is the general difference of `UNIQUE` and `PRIMARY KEY`, we'll explore the `PRIMARY KEY` in later posts.

SQLite SQL: NOT NULL column constraint

Mon, 08 Sep 2025 00:00:00 UTC

## NOT NULL Column constraint The `NOT NULL` constraint is a column-level constraint that will prevent the insertion/updation of a `NULL` value in that column record. What is `NULL`? In SQLite, specifically, a `NULL` is a value indicating that a value is not known or just not there. > NOTE: NULL is not equal to `''` empty string, `0` or `0.0` (zero), it's just cannot be compared to any value. Some additional notes on the `NOT NULL` constraint: - If a column has a `NOT NULL` constraint, and it doesn't have any other constraint like `DEFAULT`, `GENERATED`, `PRIMARY KEY`, or `FOREIGN KEY`, and there is no provided value, it will fail the constraint (i.e. error out the query). - Only if any of the other constraint populate the column while inserting the `NOT NULL` constraint will be satisfied. So, to take a simple iterative example, let's take the `users` table. ```sql CREATE TABLE users( name TEXT NOT NULL, age INTEGER CHECK(age >= 0) NOT NULL, username TEXT UNIQUE NOT NULL, language TEXT DEFAULT 'en', country_code TEXT DEFAULT 'US' NOT NULL, email TEXT UNIQUE GENERATED ALWAYS AS (username || '@myapp.com') ); ``` Here we have 5 fields - `name` which cannot be empty(`NULL`) - `age` which cannot be less than 0 or `NULL` - `username` which needs to be unique and cannot be `NULL` - `language` which has a default value of `en` but it can be `NULL` - `country_code` which has a default value of `US` but it cannot be `NULL` - `email` which has a default value of `username || '@myapp.com'` and it will be unique. Let's look at a few examples: ### Correct values ```sql INSERT INTO users (name, age, username) VALUES ('Alice', 25, 'alice'); SELECT * FROM users; ``` ### Duplicate username Won't allow duplicate username and `NULL` name fields ```sql INSERT INTO users (age, username) VALUES (25, 'alice'); SELECT * FROM users; ``` ### Border case for age Just par case for the `age` field as `0` ```sql INSERT INTO users (name, age, username) VALUES ('Bob', 0, 'bob'); SELECT * FROM users; ``` ### Invalid age check constraint The `age` is inserted as `-1` which will fail the `CHECK` constraint. ```sql INSERT INTO users (name, age, username) VALUES ('Bob', -1, 'bob'); SELECT * FROM users; ``` ### Age as null The `age` is inserted as `NULL` which will fail the `NOT NULL` constraint ```sql INSERT INTO users (name, username) VALUES ('Bob', 'bob'); SELECT * FROM users; ``` ### Correct Values Will correctly populate the values. ```sql INSERT INTO users (name, age, username) VALUES ('John', 30, 'john'); INSERT INTO users (name, age, username) VALUES ('Jim', 18, 'jim'); ``` ### Duplicate email The email inserted will lead to duplicate and fail the `UNIQUE` constraint. ```sql INSERT INTO users (name, age, username, language) VALUES ('Jim', 18, 'jim', 'fr'); ``` ### Different country code Will correctly populate the values. ```sql INSERT INTO users (name, age, username, language) VALUES ('Jimmy', 18, 'jimmy', 'fr'); ``` So, that is some combination of `NOT NULL` constraints along with other constraints and the insert values. ``` sqlite> INSERT INTO users (name, age, username) VALUES ('Alice', 25, 'alice'); sqlite> .mode table sqlite> SELECT * FROM users; +-------+-----+----------+----------+--------------+-----------------+ | name | age | username | language | country_code | email | +-------+-----+----------+----------+--------------+-----------------+ | Alice | 25 | alice | en | US | alice@myapp.com | +-------+-----+----------+----------+--------------+-----------------+ sqlite> INSERT INTO users (age, username) VALUES (25, 'alice'); Runtime error: NOT NULL constraint failed: users.name (19) sqlite> INSERT INTO users (name, age, username) VALUES ('Bob', 0, 'bob'); sqlite> SELECT * FROM users; +-------+-----+----------+----------+--------------+-----------------+ | name | age | username | language | country_code | email | +-------+-----+----------+----------+--------------+-----------------+ | Alice | 25 | alice | en | US | alice@myapp.com | | Bob | 0 | bob | en | US | bob@myapp.com | +-------+-----+----------+----------+--------------+-----------------+ sqlite> INSERT INTO users (name, age, username) VALUES ('Bob', -1, 'bob'); Runtime error: CHECK constraint failed: age >= 0 (19) sqlite> INSERT INTO users (name, username) VALUES ('Bob', 'bob'); Runtime error: NOT NULL constraint failed: users.age (19) sqlite> INSERT INTO users (name, age, username) VALUES ('John', 30, 'john'); sqlite> INSERT INTO users (name, age, username) VALUES ('Jim', 18, 'jim'); sqlite> INSERT INTO users (name, age, username, language) VALUES ('Jim', 18, 'jim', 'fr'); Runtime error: UNIQUE constraint failed: users.email (19) sqlite> INSERT INTO users (name, age, username, language) VALUES ('Jimmy', 18, 'jimmy', 'fr'); +-------+-----+----------+----------+--------------+-----------------+ | name | age | username | language | country_code | email | +-------+-----+----------+----------+--------------+-----------------+ | Alice | 25 | alice | en | US | alice@myapp.com | | Bob | 0 | bob | en | US | bob@myapp.com | | John | 30 | john | en | US | john@myapp.com | | Jim | 18 | jim | en | US | jim@myapp.com | | Jimmy | 18 | jimmy | fr | US | jimmy@myapp.com | +-------+-----+----------+----------+--------------+-----------------+ sqlite> INSERT INTO users (name, age, username) VALUES ('Robin', 24, 'robin'); sqlite> INSERT INTO users (name, age, username) VALUES ('Robinson', 24, 'robin'); Runtime error: UNIQUE constraint failed: users.email (19) sqlite> SELECT * FROM users; +-------+-----+----------+----------+--------------+-----------------+ | name | age | username | language | country_code | email | +-------+-----+----------+----------+--------------+-----------------+ | Alice | 25 | alice | en | US | alice@myapp.com | | Bob | 0 | bob | en | US | bob@myapp.com | | John | 30 | john | en | US | john@myapp.com | | Jim | 18 | jim | en | US | jim@myapp.com | | Jimmy | 18 | jimmy | fr | US | jimmy@myapp.com | | Robin | 24 | robin | en | US | robin@myapp.com | +-------+-----+----------+----------+--------------+-----------------+ sqlite> ```

SQLite SQL: Create Table with column constraints

Sun, 07 Sep 2025 00:00:00 UTC

## Create Table with column constraints We have seen the table creation with column types, we now need to look into the constraints or limitations that we can add to those columns, in order to make the data-model more robust and add some basic validations before doing any updation/mutation on the data itself. To add a constraint, we can specify it at the end of the column name, after the type, so there are a list of constraints that could be added to column, which are relevant to specific scenarios of the actual model of the column. - `NOT NULL`: This will prevent insertion of `NULL` or empty values in the column field, even updating an existing filled value with `NULL` will prevent it from inserting a `NULL` value. - `UNIQUE`: This will prevent insertion of duplicate value for an column field, same applies for updating an existing value with a duplicate will lead to violating the constraint. - `DEFAULT`: This will add a default value to the column field, if not specified in the insertion of the value. - `GENERATED ALWAYS AS`: A expression used to generate the value of the column field, this can be derived from the existing columns, or a constant value. - `CHECK`: A custom check (numeric expression) can be added for that column before updating or inserting its value in the column. - `PRIMARY KEY`: This will set the column as the primary identifier for the table, so essentially it will be like a `rowid` for the table, acting like a unique identifier among the other rows. - `FOREIGN KEY`: This will set the column as a foreign key, and it will be a reference to the primary key of another table. Let's look at a basic example of each, I would go into details in each subsequent posts in the future as their is a lot of nitty-gritty details to be covered. ## NOT NULL Constraint The `NOT NULL` constraint is a column-level constraint that can be added to a column in order to avoid getting a `NULL` or empty value being populated/inserted/updated in that column field for any row in the table. ```sql CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT, model_code TEXT ); ``` ```sql INSERT INTO llm_models DEFAULT VALUES; SELECT * FROM llm_models; ``` OOPs! `Runtime error: NOT NULL constraint failed: llm_models.name (19)`. The default value is `NULL` or empty if not explicitly constrained on the column. So, it tried to insert the `NULL` as the `name` of the `llm_models` record/row, and while doing so sqlite hit a error, at runtime, that the constraint of the column `llm_models.name` cannot be `NULL`. So, one of the solution to avoid this, could be explicitly passing a value, or setting a default value in the column setting (we will see that in the later section as the default constraint). Let's sepcify the value of the `name` column: ```sql INSERT INTO llm_models(name) VALUES('abc'); SELECT * FROM llm_models; ``` This time it successfully ran, the thing to note here is that the `model_type` or `model_code` column is populated as `NULL` becuase they don't have any constraint, especially they don't have the `NOT NULL` constraint, and hence we need to provide a value to the `name` column but the `model_type` or `model_code` columns becomes optional. SQLite also treats `''` empty string as not null, i.e. you can insert `''` into the `name` column and it will work fine. This is a bit wired but again a flexibility being a double edged sword. We need to understand what is the difference between `NULL` and empty string in this case, those are both different values. ## UNIQUE Constraint Now, we will add a constraint to the `model_code` column that it should be unique, we don't want it to be a primary key just yet, but each row/record in this table should have a unique model_code. We cannot alter the table to add a constraint in sqlite, but is allowed in other databases like postgres, mysql, etc. however that is a bit risky as the previous data might become redundant and also would fail the constraints updated. So ideally we need to re-create the table, you can however alter table columns with a combination of constraints in order to maintain the backwards data correction. Let's drop the table `llm_models` so that we can start afresh. ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT, model_code TEXT UNIQUE ); ``` This way, we make sure the `model_code` column is unique, and we don't have any duplicate values in the column. When each new record is added to the table `llm_models` the `model_code` needs to be unique and even updation would fail if the new value is duplicated. ```sql INSERT INTO llm_models(name) VALUES('abc'); SELECT * FROM llm_models; ``` Works fine, now notice here that the `model_code` is not provided and therefore it is `NULL`, let's add one more row with the same `name` and check if that fails the unique constraint on the `model_code` as it will now have the model_code duplicated `NULL` in both the rows, that should fail right? RIGHT? ```sql INSERT INTO llm_models(name) VALUES('abc'); SELECT * FROM llm_models; ``` Nopes! `NULL` is not a value to be called as unique. NULL is like undefined, you can't really distinguish between one `NULL` and other `NULL`s. Hence, the constraint is often associated with the `NOT NULL` as well to prevent the `NULL` confusion. Let's add the `model_code` now. ```sql INSERT INTO llm_models(name, model_code) VALUES ('abc', 'llm-1'); SELECT * FROM llm_models; ``` Ok, we added a record with `model_code` as `llm-1`. Now, let's add the same record again. ```sql INSERT INTO llm_models(name, model_code) VALUES ('abc', 'llm-1'); SELECT * FROM llm_models; ``` And, it broke, it gave a `Runtime error: UNIQUE constraint failed: llm_models.model_code (19)` error. So, we can't add the same `model_code` twice. ``` sqlite> CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT, model_code TEXT UNIQUE ); sqlite> INSERT INTO llm_models(name) VALUES('abc'); sqlite> .mode table sqlite> SELECT * FROM llm_models; +------+------------+------------+ | name | model_type | model_code | +------+------------+------------+ | abc | | | +------+------------+------------+ sqlite> INSERT INTO llm_models(name) VALUES('abc'); sqlite> SELECT * FROM llm_models; +------+------------+------------+ | name | model_type | model_code | +------+------------+------------+ | abc | | | | abc | | | +------+------------+------------+ sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); Runtime error: UNIQUE constraint failed: llm_models.model_code (19) sqlite> SELECT * FROM llm_models; +------+------------+------------+ | name | model_type | model_code | +------+------------+------------+ | abc | | | | abc | | | | abc | | llm-1 | +------+------------+------------+ sqlite> ``` So some tips for using the `UNIQUE` constraints in sqlite: - You can't use `UNIQUE` constraint on the `NULL` values. - Prefer using `NOT NULL` in addition to the `UNIQUE` constraint. ## DEFAULT Constraint The default constraint as the name suggest, will set a default value for a column if no value is provided in the insert statement. However, you need to provide the default value while creating the table. You can generate a default value (for that we can use the `GENERATED ALWAYS` constraint). ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT DEFAULT 'text', model_code TEXT UNIQUE NOT NULL ); ``` I have added a constraint of `model_type` as `DEFAULT 'text'` which will set the value of `model_type` column to `text` if no value is provided in the insert statement. Also, the `model_code` column is updated with the constraints of `UNIQUE` and `NOT NULL` ```sql INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); SELECT * FROM llm_models; ``` And that works as expected, it populates the value of `model_type` as `text` by default. ``` sqlite> CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT DEFAULT 'text', model_code TEXT UNIQUE NOT NULL ); sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); sqlite> SELECT * FROM llm_models; +------+------------+------------+ | name | model_type | model_code | +------+------------+------------+ | abc | text | llm-1 | +------+------------+------------+ sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); Runtime error: UNIQUE constraint failed: llm_models.model_code (19) sqlite> ``` And if we try to insert the same `model_code` value again, it will fail with the `UNIQUE` constraint. ## Generated Constraint The [Generated](https://sqlite.org/gencol.html) constraint can add a value to a column based on evaluated expression from the values of the other columns in that row. ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT DEFAULT 'text', model_code TEXT UNIQUE NOT NULL, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED ); ``` We have added a `GENERATED ALWAYS AS STORED` constraint which will store the result of `name || ' (' || model_code || ')'` in the `display_name` column. ```sql INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); SELECT * FROM llm_models; ``` There is a variant of `VIRTUAL` instead of `STORED`, we will explore this separately, I am just getting my hands on and getting familiarity with all the constraints available in SQL. ``` sqlite> CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT DEFAULT 'text', model_code TEXT UNIQUE NOT NULL, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED ); sqlite> .mode table sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); sqlite> SELECT * FROM llm_models; +------+------------+------------+--------------+ | name | model_type | model_code | display_name | +------+------------+------------+--------------+ | abc | text | llm-1 | abc (llm-1) | +------+------------+------------+--------------+ ``` Neat! It populated, or I should say `generated` the `display_name` column by evaluating the `name || ' (' || model_code || ')'` expression. ## Check Constraint The `CHECK` constraint is a column-level constraint that can be added to a column in order to add a custom check (numeric expression) before updating or inserting its value in the column. Just a simple expression or a validation check before inserting the value to the column in the row. Let's take the `model_type` constraint with the `CHECK` constraint by limiting the values of `model_type` to `text`, `conversational`, `multimodal`, `code`. ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED ); ``` Here, we have added the `CHECK` constraint to the `model_type` column. And we have limited the values of `model_type` to `text`, `conversational`, `multimodal`, `code`. It could be anything, but its just a true or false check that will either fail or work. ```sql INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); SELECT * FROM llm_models; ``` This will add the default value as `text` for the `model_type`. ```sql INSERT INTO llm_models(name, model_code, model_type) VALUES('abc', 'llm-1', 'visual'); SELECT * FROM llm_models; ``` Here we try to add the column `model_type` as `visual` which is not allowed by the `CHECK` constraint. ```sql INSERT INTO llm_models(name, model_code, model_type) VALUES('abc', 'llm-1', 'code'); SELECT * FROM llm_models; ``` And we try to add the column `model_type` as `code` which is not allowed by the `CHECK` constraint. ``` sqlite> CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED ); sqlite> INSERT INTO llm_models(name, model_code) VALUES('abc', 'llm-1'); sqlite> SELECT * FROM llm_models ...> ; +------+------------+------------+--------------+ | name | model_type | model_code | display_name | +------+------------+------------+--------------+ | abc | text | llm-1 | abc (llm-1) | +------+------------+------------+--------------+ sqlite> INSERT INTO llm_models(name, model_code, model_type) VALUES('abc', 'llm-1', 'visual'); Runtime error: CHECK constraint failed: model_type IN ('text', 'conversational', 'multimodal', 'code') (19) sqlite> INSERT INTO llm_models(name, model_code, model_type) VALUES('abc', 'llm-1', 'code'); Runtime error: UNIQUE constraint failed: llm_models.model_code (19) sqlite> INSERT INTO llm_models(name, model_code, model_type) VALUES('abc', 'llm-2', 'code'); sqlite> SELECT * FROM llm_models; +------+------------+------------+--------------+ | name | model_type | model_code | display_name | +------+------------+------------+--------------+ | abc | text | llm-1 | abc (llm-1) | | abc | code | llm-2 | abc (llm-2) | +------+------------+------------+--------------+ sqlite> ``` Right! We checked that it failed the constraint `model_type IN ('text', 'conversational', 'multimodal', 'code')` and we checked that it failed the constraint `UNIQUE` constraint. ## Primary Key Constraint The primary key constraint is a column-level constraint that is used to identify a unique row in a table. It is a unique constraint that can be used to identify a row in a table using a single column. We can provide it right after the column name or at the bottom of all the table constraints. ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED, PRIMARY KEY(model_code) ); ``` OR ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL PRIMARY KEY, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED ); ``` The `PRIMARY KEY` is a table level constraint, as it needs to unique across all the rows in the table. ## Foreign Key Constraint The foreign key constraint is a column-level constraint that is used to establish a relationship between two tables. We will now create a separate table called `companies` for this example, this will be a table that stores the company information those will be creating or developing the LLM models. ```sql CREATE TABLE companies( id INTEGER PRIMARY KEY, company_name TEXT UNIQUE NOT NULL, founded_year INTEGER ); ``` Let's insert a few companies ```sql INSERT INTO companies (company_name, founded_year) VALUES('ABC', 1998); INSERT INTO companies (company_name, founded_year) VALUES('ClosedAI', 2016); INSERT INTO companies (company_name, founded_year) VALUES('Beta', 2014); SELECT * FROM companies; ``` Original companies right? Now we will update the `llm_models` table that will store the LLM models information. We will add a foreign key, that will be a reference i.e. to make a relation for the row that will reference another row. So, the foreign key indicates a key (field/column) in another table that's why named as `FOREIGN KEY`. This foreign key will be referencing the `companies.id` column from the `companies` table. ```sql DROP TABLE llm_models; CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL PRIMARY KEY, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED, company_id INTEGER, FOREIGN KEY(company_id) REFERENCES companies(id) ); ``` We insert the `llm_models` with the foreign key of 2 indicating the company `ClosedAI`. ```sql INSERT INTO llm_models(name, model_code, company_id) VALUES('abc', 'llm-1', 2); SELECT * FROM llm_models; ``` Now, if we see, this has successfully inserted the row. ```sql SELECT * FROM llm_models; ``` We can join the two tables together, this will give us the combined sets form the tables. ```sql SELECT * FROM llm_models as llm INNER JOIN companies as comp ON comp.id = llm.company_id; ``` That was a bit unclear, let's only keep what we needed. ```sql SELECT llm.*, comp.company_name FROM llm_models as llm INNER JOIN companies as comp ON comp.id = llm.company_id; ``` Cool, that is sweat. ``` sqlite> CREATE TABLE companies( id INTEGER PRIMARY KEY, company_name TEXT UNIQUE NOT NULL, founded_year INTEGER ); sqlite> INSERT INTO companies (company_name, founded_year) VALUES('ABC', 1998); INSERT INTO companies (company_name, founded_year) VALUES('ClosedAI', 2016); INSERT INTO companies (company_name, founded_year) VALUES('Beta', 2014); SELECT * FROM companies; +----+--------------+--------------+ | id | company_name | founded_year | +----+--------------+--------------+ | 1 | ABC | 1998 | | 2 | ClosedAI | 2016 | | 3 | Beta | 2014 | +----+--------------+--------------+ sqlite> sqlite> CREATE TABLE llm_models( name TEXT NOT NULL, model_type TEXT CHECK(model_type IN ('text', 'conversational', 'multimodal', 'code')) DEFAULT 'text', model_code TEXT UNIQUE NOT NULL PRIMARY KEY, display_name TEXT GENERATED ALWAYS AS (name || ' (' || model_code || ')') STORED, company_id INTEGER, FOREIGN KEY(company_id) REFERENCES companies(id) ); sqlite> SELECT * FROM llm_models; sqlite> INSERT INTO llm_models(name, model_code, model_type, company_id) VALUES('abc', 'llm-1', 'code', 2); sqlite> SELECT * FROM llm_models; +------+------------+------------+--------------+------------+ | name | model_type | model_code | display_name | company_id | +------+------------+------------+--------------+------------+ | abc | code | llm-1 | abc (llm-1) | 2 | +------+------------+------------+--------------+------------+ sqlite> SELECT * FROM llm_models as llm INNER JOIN companies as comp ON comp.id = llm.company_id; +------+------------+------------+--------------+------------+----+--------------+--------------+ | name | model_type | model_code | display_name | company_id | id | company_name | founded_year | +------+------------+------------+--------------+------------+----+--------------+--------------+ | abc | code | llm-1 | abc (llm-1) | 2 | 2 | ClosedAI | 2016 | +------+------------+------------+--------------+------------+----+--------------+--------------+ sqlite> SELECT llm.*, comp.company_name FROM llm_models as llm INNER JOIN companies as comp ON comp.id = llm.company_id; +------+------------+------------+--------------+------------+--------------+ | name | model_type | model_code | display_name | company_id | company_name | +------+------------+------------+--------------+------------+--------------+ | abc | code | llm-1 | abc (llm-1) | 2 | ClosedAI | +------+------------+------------+--------------+------------+--------------+ sqlite> ``` That is it, a high level overview of adding constraints in SQLite, I will dig deep into each constraint in separate posts over the coming week.

SQLite SQL: Create Table with column types

Sat, 06 Sep 2025 00:00:00 UTC

## Creating a table with column types If you want to store values in a column with a specific type, you need to specify one of the following types in order to store that value in relatively type specific data. - `TEXT` (string or characters, bunch of text) - `BLOB` (binary data, images, raw files, etc) - `INTEGER` (numbers which are whole, like 1, 67, 986, etc) - `REAL` (floating or decimal point numbers, like 3.14, 85.98, 999.99, etc) - `NULL` (empty or not defined) Type affinity is what determines how a data is being stored and treated in sqlite, so we'll need to cover that separately. For now, let's assume that the data you give in is indeed valdiated before sending over to the database (that rarely is the case though) but for now we can avoid that discussion, and focus on creation of the structure and relations of the data. If you are curious, you can learn about types and affinity [here](https://www.sqlite.org/datatype3.html). Let's see how to create a table with a few columns having a specific type. Let's say we are creating a database system and a schema for storing LLMs and their history. How interesting! ```sql CREATE TABLE llm_models( name TEXT, model_type TEXT, description TEXT, no_parameters REAL, f1_score REAL, input_token_limit INTEGER, output_token_limit INTEGER, context_window_tokens INTEGER, release_date TEXT, knowledge_cutoff_date TEXT, weights_file BLOB, license TEXT ); ``` As you can see, we have a few columns with specific types. The name, model type, description, licence could be plain text. The number of parameters and f1_score and others could be real, as those exact number of the parameters cannot be discrete values. Similarly, the `input_token_limit`, `output_token_limit`, `context_window_tokens` can be a discrete value hence used as integer. The `release_date` and `knowledge_cutoff_date` is a date value so could be used as numeric-affinity but we need to store it as text for now. The `weights_file` is a binary file, hence storing it as a blob i.e. binary large object. So, if we were to insert into these the default values, what do you think what will it populate with? ```sql INSERT INTO llm_models DEFAULT VALUES; ``` ```sql SELECT * FROM llm_models; ``` ``` sqlite> select * from llm_models; +------+------------+-------------+---------------+-------------------+--------------------+-----------------------+--------------+-----------------------+--------------+---------+ | name | model_type | description | no_parameters | input_token_limit | output_token_limit | context_window_tokens | release_date | knowledge_cutoff_date | weights_file | license | +------+------------+-------------+---------------+-------------------+--------------------+-----------------------+--------------+-----------------------+--------------+---------+ | | | | | | | | | | | | +------+------------+-------------+---------------+-------------------+--------------------+-----------------------+--------------+-----------------------+--------------+---------+ sqlite> ``` Well all is empty! That's becuase it is `None` or `NULL` by default, and we don't have any constraint on the columns as well as on the tables. We need to add some constraints on the columns as well as the tables. There are options to add contraints on the column like - `NOT NULL` - `UNIQUE` - `PRIMARY KEY` - `CHECK` - `DEFAULT` - `COLLATE` - `FOREIGN KEY` We'll explore those separately for now understand those as some restrictions on how the column values can be stored. For instance in our example, we can't store a record or a llm model without a name so we need to have it as `NOT NULL` which will prevent a record from being inserted as empty or NULL essentially. It turns out we can also add in a `STRICT` paramter to the end of the table to make sure the table doesn't allow `TEXT` values to be stored in `INTEGER` column. There are a lot of details I think I am going to leave it right here. We only need to add these for our conveniences, sqlite is so flexible that it can be a little intimidating to add these values and not get benefit right away, but imagine this. You have a quick experiment to run and test out how the data looks like, your data model works, you just need your crud app to be ready, no time in wasting which will be the primary key, which will be a not null column, what the default value is for each column, it can be skipped with SQLite. That's a double-edged sword, yes in some sense, but more powerful for most of the applications. It assumes you don't need validations unless you specify it, and believe me there are tons of validations you can add in sqlite to make it robust.

SQLite SQL: Create Table Bare Bones Table

Fri, 05 Sep 2025 00:00:00 UTC

What if you only wanted one column. If you create specify a column name in create table, then there will be actually 2 columns created, one that you specified and the other which is hidden as `rowid`, unless you mentioned rowid as the column. ```sql CREATE TABLE users(name); INSERT INTO users(name) VALUES ("abc"), ("def"); SELECT * FROM users; ``` You would think, you have one column `name` in the table `users`. Nope! ```sql SELECT rowid, * from users; ``` ``` +-------+------+ | rowid | name | +-------+------+ | 1 | abc | | 2 | def | +-------+------+ ``` Surprise! Any user defined table in sqlite almost always has a [rowid column](https://www.sqlite.org/rowidtable.html) That tempts me to create a table with one column, and see what happens, how can we uniquely identify rows in that case. ```sql CREATE TABLE users(rowid); INSERT INTO users DEFAULT VALUES; INSERT INTO users DEFAULT VALUES; .mode table SELECT * FROM users; ``` NOTE: DEFAULT VALUES, this means, use the default values for the columns, if you didn't specify a type for the column, the default values are `NULL`. ``` sqlite> create table users(rowid); sqlite> insert into users default values; sqlite> .mode table sqlite> select * from users; +-------+ | rowid | +-------+ | | +-------+ sqlite> insert into users default values; sqlite> select * from users; +-------+ | rowid | +-------+ | | | | +-------+ sqlite> select distinct * from users; +-------+ | rowid | +-------+ | | +-------+ sqlite> select distinct rowid from users; +-------+ | rowid | +-------+ | | +-------+ ``` Here, its kind of hard to distinguish each row, so we have kind of skewed up. There is only one row and its not anymore served as a unique identifier in the table. As you can insert multiple same values for the users table. ```sql DROP TABLE users; ``` Let's create a table with a table that increments the rowid auto-incrementally. ```sql CREATE TABLE users(rowid INTEGER PRIMARY KEY AUTOINCREMENT); INSERT INTO users DEFAULT VALUES; INSERT INTO users DEFAULT VALUES; SELECT * FROM users; ``` Now it simply increments each time we add a row, that's a neat little table. I think this is far we can push sqlite, just one column. But wait, what's beneath this? Like why rowid? Why not name? Well, SQLite always creates this underlying rowid for structural reasons. When we name our column rowid, we're just making that hidden rowid visible. But if you create an `INTEGER PRIMARY KEY` (regardless of its name), that becomes an alias for the underlying `rowid` and not just any primary key, specifically `INTEGER PRIMARY KEY` But here, we ourselves defined the rowid, so when you create something like a `id` not `rowid` as a primary key, it will be a alias for that underlying `rowid` In the first example, the `CREATE TABLE users(rowid)` this is a violation of sqlite, as we now have created a table which cannot be distinctly identified with a column, as there is only one column, and that too is exposed to the user, and hence in the default case, the `rowid` is hidden column, unless the user/developer has added a primary key to a certain row. The user/developer is taking away the ownership of managing the primary key (unique identifier for a row/entry in a table) to him/herself rather than relying on the `rowid` solely.

SQLite SQL: Create Table Basic

Thu, 04 Sep 2025 00:00:00 UTC

We have explored enough sqlite dot commands, now I want to explore the actual nuts and bolts of SQL in SQLite, what better place then to create tables. ## CREATE TABLE Statement The `CREATE TABLE` as the name suggest will create a table in a sqlite database. What is a table? > Table is a set of relations The syntax for the create table might look something like this: ```sql CREATE [TEMP | TEMPORARY] TABLE [IF NOT EXISTS] [schema-name.]table-name ( column-def ( , column-def )* [, table-constraint]* ) [WITHOUT ROWID] [STRICT] ``` Check [this diagram](https://sqlite.org/lang_createtable.html) to make things clear for you, might even confuse you if you are absolutely new, so just stick around we will explore it one piece at a time, and as usual follow a exhaustive, all combination exploration of this statemnet. There are a lot of things, but we'll see each of them one by one, not all at once. The important and mandatory things to provide are table name and atleast one column name. Not even the type of the column is needed, just the name is sufficient, since the type is really not a thing in sqlite tables unless you add `STRICT` as the table option. So, the bare bones table creation statement might look something like this: ```sql CREATE TABLE users(name); ``` Dead! Just the table name and one column name, can you even go with no columns (Except the rowid one that is automagically added?). You can actually, but you will have to specify the column as rowid (let’s keep that aside for now). This is wired, but was juse exploring the possibilities of what the bare minimum is required for creating a table in sqlite. Now, if we want to insert this into the `users` table, we can simply do this: ```sql INSERT INTO users(name) VALUES ("abc"), ("def"), (100), (89.8); SELECT * FROM users; ``` Simple right? Note that you can insert anything in the id column, not just numbers, but you can't insert anything in the name column, since that is a mandatory column. That is wired but that's what sqlite has a duct typing if not opted out of it.

SQLite dot commands: header command

Wed, 03 Sep 2025 00:00:00 UTC

The `.headers` command is crucial as might effect the way the result set is displayed. The first row would be set to the name of the columns in the output of the relevant commands like `.output`, `.once`, or even your simple standard out queries if it is set on. ``` .headers on OR .header on ``` Some how either of them work. You need to set it to `on` to enable the headers in the output of the result set. And you can turn off with `.headers off`. Which modes are effected with this command if set on or off? Well we need to think about how effected means 1. Only added in the first row 2. Added in each row 3. No effect --- 1. Only added in the first row 1. ascii 2. column 3. csv 4. html 5. list 6. quote 7. tabs 8. tcl 2. Added in each row 1. insert 3. No effect 1. box 2. json 3. line 4. markdown 5. table Reference table ```sql CREATE TABLE IF NOT EXISTS books( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT NOT NULL, author TEXT NOT NULL, pages INTEGER NOT NULL, release_date TEXT NOT NULL, price REAL NOT NULL ); INSERT INTO books(title, author, pages, release_date, price) VALUES ('The Hobbit', 'J.R.R. Tolkien', 310, '1937-09-21', 39.99), ('The Fellowship of the Ring', 'J.R.R. Tolkien', 423, '1954-07-29', 49.99), ('The Two Towers', 'J.R.R. Tolkien', 352, '1954-11-11', 49.99), ('The Return of the King', 'J.R.R. Tolkien', 416, '1955-10-20', 49.99); SELECT * FROM books; ``` Below are the outputs of each command ### Added on the first row In the below modes, the headers command if set will add the column name as the header in the first row. 1. ASCII ```sqlite .mode ascii .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode ascii sqlite> .headers on sqlite> select * from books; idtitleauthorpagesrelease_dateprice1The HobbitJ.R.R. Tolkien3101937-09-2139.992The Fellowship of the RingJ.R.R. Tolkien4231954-07-2949.993The Two TowersJ.R.R. Tolkien3521954-11-1149.994The Return of the KingJ.R.R. Tolkien4161955-10-2049.99sqlite> sqlite> .headers off sqlite> select * from books; 1The HobbitJ.R.R. Tolkien3101937-09-2139.992The Fellowship of the RingJ.R.R. Tolkien4231954-07-2949.993The Two TowersJ.R.R. Tolkien3521954-11-1149.994The Return of the KingJ.R.R. Tolkien4161955-10-2049.99sqlite> sqlite> ``` 2. Column ``` .mode column .headers on select * from books; .headers off select * from books; ``` Output ``` sqlite> .mode column sqlite> .headers on sqlite> select * from books; id title author pages release_date price -- -------------------------- -------------- ----- ------------ ----- 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> .headers off sqlite> select * from books; 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> ``` 3. CSV ```sqlite .mode csv .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode csv sqlite> .headers on sqlite> select * from books; id,title,author,pages,release_date,price 1,"The Hobbit","J.R.R. Tolkien",310,1937-09-21,39.99 2,"The Fellowship of the Ring","J.R.R. Tolkien",423,1954-07-29,49.99 3,"The Two Towers","J.R.R. Tolkien",352,1954-11-11,49.99 4,"The Return of the King","J.R.R. Tolkien",416,1955-10-20,49.99 sqlite> .headers off sqlite> select * from books; 1,"The Hobbit","J.R.R. Tolkien",310,1937-09-21,39.99 2,"The Fellowship of the Ring","J.R.R. Tolkien",423,1954-07-29,49.99 3,"The Two Towers","J.R.R. Tolkien",352,1954-11-11,49.99 4,"The Return of the King","J.R.R. Tolkien",416,1955-10-20,49.99 sqlite> ``` 4. HTML ```sqlite .mode html .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode html sqlite> .headers on sqlite> select * from books; id title author pages release_date price 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> .headers off sqlite> select * from books; 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> ``` 5. List ```sqlite .mode list .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode list sqlite> .headers on sqlite> select * from books; id|title|author|pages|release_date|price 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99 2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99 3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99 4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99 sqlite> .headers off sqlite> select * from books; 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99 2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99 3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99 4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99 sqlite> ``` 6. Quote ```sqlite .mode quote .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode quote sqlite> .headers on sqlite> select * from books; 'id','title','author','pages','release_date','price' 1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198 2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198 3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198 4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198 sqlite> .headers off sqlite> select * from books; 1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198 2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198 3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198 4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198 sqlite> ``` 7. Tabs ```sqlite .mode tabs .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode tabs sqlite> .headers on sqlite> select * from books; id title author pages release_date price 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> .headers off sqlite> select * from books; 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> ``` 8. TCL ```sqlite .mode tcl .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode tcl sqlite> .headers on sqlite> select * from books; "id" "title" "author" "pages" "release_date" "price" "1" "The Hobbit" "J.R.R. Tolkien" "310" "1937-09-21" "39.99" "2" "The Fellowship of the Ring" "J.R.R. Tolkien" "423" "1954-07-29" "49.99" "3" "The Two Towers" "J.R.R. Tolkien" "352" "1954-11-11" "49.99" "4" "The Return of the King" "J.R.R. Tolkien" "416" "1955-10-20" "49.99" sqlite> .headers off sqlite> select * from books; "1" "The Hobbit" "J.R.R. Tolkien" "310" "1937-09-21" "39.99" "2" "The Fellowship of the Ring" "J.R.R. Tolkien" "423" "1954-07-29" "49.99" "3" "The Two Towers" "J.R.R. Tolkien" "352" "1954-11-11" "49.99" "4" "The Return of the King" "J.R.R. Tolkien" "416" "1955-10-20" "49.99" sqlite> ``` ### Added in each row In this type, each row has the column name as some form of the row. 1. Insert ```sqlite .mode insert .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode insert sqlite> .headers on sqlite> select * from books; INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198); sqlite> .headers off sqlite> select * from books; INSERT INTO "table" VALUES(1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198); INSERT INTO "table" VALUES(2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198); INSERT INTO "table" VALUES(3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198); INSERT INTO "table" VALUES(4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198); sqlite> ``` ### No effect These modes show the headers or column names irrespective of the `.headers` flag, as for these modes the headers form quite a bit of structure to the output they are showing. 1. Box ```sqlite .mode box .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode box sqlite> .headers on sqlite> select * from books; ┌────┬────────────────────────────┬────────────────┬───────┬──────────────┬───────┐ │ id │ title │ author │ pages │ release_date │ price │ ├────┼────────────────────────────┼────────────────┼───────┼──────────────┼───────┤ │ 1 │ The Hobbit │ J.R.R. Tolkien │ 310 │ 1937-09-21 │ 39.99 │ │ 2 │ The Fellowship of the Ring │ J.R.R. Tolkien │ 423 │ 1954-07-29 │ 49.99 │ │ 3 │ The Two Towers │ J.R.R. Tolkien │ 352 │ 1954-11-11 │ 49.99 │ │ 4 │ The Return of the King │ J.R.R. Tolkien │ 416 │ 1955-10-20 │ 49.99 │ └────┴────────────────────────────┴────────────────┴───────┴──────────────┴───────┘ sqlite> .headers off sqlite> select * from books; ┌────┬────────────────────────────┬────────────────┬───────┬──────────────┬───────┐ │ id │ title │ author │ pages │ release_date │ price │ ├────┼────────────────────────────┼────────────────┼───────┼──────────────┼───────┤ │ 1 │ The Hobbit │ J.R.R. Tolkien │ 310 │ 1937-09-21 │ 39.99 │ │ 2 │ The Fellowship of the Ring │ J.R.R. Tolkien │ 423 │ 1954-07-29 │ 49.99 │ │ 3 │ The Two Towers │ J.R.R. Tolkien │ 352 │ 1954-11-11 │ 49.99 │ │ 4 │ The Return of the King │ J.R.R. Tolkien │ 416 │ 1955-10-20 │ 49.99 │ └────┴────────────────────────────┴────────────────┴───────┴──────────────┴───────┘ sqlite> ``` 2. JSON ```sqlite .mode json .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode json sqlite> .headers on sqlite> select * from books; [{"id":1,"title":"The Hobbit","author":"J.R.R. Tolkien","pages":310,"release_date":"1937-09-21","price":39.99000000000000198}, {"id":2,"title":"The Fellowship of the Ring","author":"J.R.R. Tolkien","pages":423,"release_date":"1954-07-29","price":49.99000000000000198}, {"id":3,"title":"The Two Towers","author":"J.R.R. Tolkien","pages":352,"release_date":"1954-11-11","price":49.99000000000000198}, {"id":4,"title":"The Return of the King","author":"J.R.R. Tolkien","pages":416,"release_date":"1955-10-20","price":49.99000000000000198}] sqlite> .headers off sqlite> select * from books; [{"id":1,"title":"The Hobbit","author":"J.R.R. Tolkien","pages":310,"release_date":"1937-09-21","price":39.99000000000000198}, {"id":2,"title":"The Fellowship of the Ring","author":"J.R.R. Tolkien","pages":423,"release_date":"1954-07-29","price":49.99000000000000198}, {"id":3,"title":"The Two Towers","author":"J.R.R. Tolkien","pages":352,"release_date":"1954-11-11","price":49.99000000000000198}, {"id":4,"title":"The Return of the King","author":"J.R.R. Tolkien","pages":416,"release_date":"1955-10-20","price":49.99000000000000198}] sqlite> ``` 3. Line ```sqlite .mode line .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode line sqlite> .headers on sqlite> select * from books; id = 1 title = The Hobbit author = J.R.R. Tolkien pages = 310 release_date = 1937-09-21 price = 39.99 id = 2 title = The Fellowship of the Ring author = J.R.R. Tolkien pages = 423 release_date = 1954-07-29 price = 49.99 id = 3 title = The Two Towers author = J.R.R. Tolkien pages = 352 release_date = 1954-11-11 price = 49.99 id = 4 title = The Return of the King author = J.R.R. Tolkien pages = 416 release_date = 1955-10-20 price = 49.99 sqlite> .headers off sqlite> select * from books; id = 1 title = The Hobbit author = J.R.R. Tolkien pages = 310 release_date = 1937-09-21 price = 39.99 id = 2 title = The Fellowship of the Ring author = J.R.R. Tolkien pages = 423 release_date = 1954-07-29 price = 49.99 id = 3 title = The Two Towers author = J.R.R. Tolkien pages = 352 release_date = 1954-11-11 price = 49.99 id = 4 title = The Return of the King author = J.R.R. Tolkien pages = 416 release_date = 1955-10-20 price = 49.99 sqlite> ``` 4. Markdown ```sqlite .mode markdown .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode markdown sqlite> .headers on sqlite> select * from books; | id | title | author | pages | release_date | price | |----|----------------------------|----------------|-------|--------------|-------| | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | sqlite> .headers off sqlite> select * from books; | id | title | author | pages | release_date | price | |----|----------------------------|----------------|-------|--------------|-------| | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | sqlite> ``` 5. Table ```sqlite .mode table .headers on select * from books; .headers off select * from books; ``` Output: ``` sqlite> .mode table sqlite> .headers on sqlite> select * from books; +----+----------------------------+----------------+-------+--------------+-------+ | id | title | author | pages | release_date | price | +----+----------------------------+----------------+-------+--------------+-------+ | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | +----+----------------------------+----------------+-------+--------------+-------+ sqlite> .headers off sqlite> select * from books; +----+----------------------------+----------------+-------+--------------+-------+ | id | title | author | pages | release_date | price | +----+----------------------------+----------------+-------+--------------+-------+ | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | +----+----------------------------+----------------+-------+--------------+-------+ sqlite> ```

SQLite dot commands: Output mode separator command

Tue, 02 Sep 2025 00:00:00 UTC

## Using the separator for the ouput If you wanted to use a specific separator for columns and rows while displaying the result set / table, you can use the `.separator` dot command which can take 2 arguments, first as the separator for the column and the second for the row. So, if we set use `.separator "|" "---"` then it will split the columns with `|` and each row with `---`. ``` 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99---2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99---3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99---4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99--- ``` The output looks wired but I was giving a example. The row separator is by default a `\n` character or `\r\n` on windows, which is for the list or any other mode. However if you want to add those, you need to specify it in the string like below: ``` .separator "|" "\n---" ``` ``` >sqlite>.separator "|" "\n---" sqlite> select * from books; 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99 ---2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99 ---3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99 ---4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99 ---sqlite> ``` The `\n---` will add it at the end of each row, so the first row won't have it if you are seeing the above output and confused, then that's the expected result. You can also use `\r` or `\t` for the row separator, and for the column separator you can use `\n` or `\t` or `\r`. The modes that will respect the separator setting: - list - quote Sadly only two, since for changing the row separator you need to pass the column separator first, so that defeats the purpose of the change in the row separator, you'll have to override the column separator first and then the row. So, for the mode `tabs`, if you want to keep the columns separator as is, but change the row separator then you can do something like this: ``` .mode tabs .separator "\t" "\n\n" ``` This will keep the column separator as is and change the row separator to `\n\n` ``` sqlite> .mode tabs sqlite> .separator "\t" "\n\n" sqlite> select * from books; id title author pages release_date price 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 sqlite> ``` Similarly for csv you would keep the separator as , and then pass the row separator Nice tricks to know. References: - [SQLite dot commands: Changing output formats](https://www.sqlite.org/cli.html#changing_output_formats)

SQLite dot commands: Output mode

Mon, 01 Sep 2025 00:00:00 UTC

## Output mode dot command The SQLite shell is a great way to test out and run bunch of queries, but the output might be squished or cluttered. You might need to view the output in a specific way, and the creators of SQLite were already hearing your screams. They created not 5, not 10, but 14 different output modes, and to extend it even further, you can even use any delimiter/separator as well. The `.mode` will give you the currently configured/set output mode with some options which are default for certain type of modes. The default mode is `list`, with the option of `escape` with the ascii character set. ``` current output mode: list --escape ascii ``` The list mode defined in the [docs](https://www.sqlite.org/cli.html#changing_output_formats) as: > In list mode, each row of a query result is written on one line of output and each column within that row is separated by a specific separator string. The default separator is a pipe symbol ("|"). List mode is especially useful when you are going to send the output of a query to another program (such as AWK) for additional processing. So, in short, the `list` output mode prints each row on a line with a `|` as the separator. You can get all the information you need with the `.help mode` command ``` .mode ?MODE? ?OPTIONS? Set output mode MODE is one of: ascii Columns/rows delimited by 0x1F and 0x1E box Tables using unicode box-drawing characters csv Comma-separated values column Output in columns. (See .width) html HTML \ code insert SQL insert statements for TABLE json Results in a JSON array line One value per line list Values delimited by "|" markdown Markdown table format qbox Shorthand for "box --wrap 60 --quote" quote Escape answers as for SQL table ASCII-art table tabs Tab-separated values tcl TCL list elements OPTIONS: (for columnar modes or insert mode): --escape T ctrl-char escape; T is one of: symbol, ascii, off --wrap N Wrap output lines to no longer than N characters --wordwrap B Wrap or not at word boundaries per B (on/off) --ww Shorthand for "--wordwrap 1" --quote Quote output text as SQL literals --noquote Do not quote output text TABLE The name of SQL table used for "insert" mode ``` Let's set up with some sample data ```sql CREATE TABLE IF NOT EXISTS books( id INTEGER PRIMARY KEY AUTOINCREMENT, title TEXT NOT NULL, author TEXT NOT NULL, pages INTEGER NOT NULL, release_date TEXT NOT NULL, price REAL NOT NULL ); INSERT INTO books(title, author, pages, release_date, price) VALUES ('The Hobbit', 'J.R.R. Tolkien', 310, '1937-09-21', 39.99), ('The Fellowship of the Ring', 'J.R.R. Tolkien', 423, '1954-07-29', 49.99), ('The Two Towers', 'J.R.R. Tolkien', 352, '1954-11-11', 49.99), ('The Return of the King', 'J.R.R. Tolkien', 416, '1955-10-20', 49.99); SELECT * FROM books; ``` ``` 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99 2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99 3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99 4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99 ``` Not the best way to look at the data, but handy in some cases. We can change it, we have 14 different modes to choose from. Its like choosing a taste flovour for an ice-cream, you want something different each time, or you stick to the simple vanilla one. Which one are you? I am a vanilla guy (I like the table mode, but use csv heavily too). Let's change it to a table format with `.mode table` This will set the mode as `table` > table ASCII-art table This will show the result sets in a ascii-like table structure with the `+++` and `---` to separate the rows and columns. ``` SELECT * FROM books; ``` ``` +----+----------------------------+----------------+-------+--------------+-------+ | id | title | author | pages | release_date | price | +----+----------------------------+----------------+-------+--------------+-------+ | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | +----+----------------------------+----------------+-------+--------------+-------+ ``` That is soo nice looking! Crystal clear. You can even see the current set mode precisely with the `.mode` command Since I changed the mode to table, let's see what the mode is now set as? ``` sqlite> .mode current output mode: table --wrap 60 --wordwrap off --noquote --escape ascii ``` It has a bunch of options like `wrap`, `wordwrap`, `noquote`, and `escape` etc. You can take a look at the help and tweak them as per your liking, I usually don't change the options, but it might be very rare to switch from the defaults. ## Output modes ### ASCII This mode separates the columns with by `0x1F` and rows with `0x1E`. ``` .mode ascii SELECT * FROM books; ``` ``` id�title�author�pages�release_date�price 1�The Hobbit�J.R.R. Tolkien�310�1937-09-21�39.99 2�The Fellowship of the Ring�J.R.R. Tolkien�423�1954-07-29�49.99 3�The Two Towers�J.R.R. Tolkien�352�1954-11-11�49.99 4�The Return of the King�J.R.R. Tolkien�416�1955-10-20�49.99 ``` ### Box The box mode renders the result set using unicode box-drawing characters. ``` .mode box SELECT * FROM books; ``` ``` ┌────┬────────────────────────────┬────────────────┬───────┬──────────────┬───────┐ │ id │ title │ author │ pages │ release_date │ price │ ├────┼────────────────────────────┼────────────────┼───────┼──────────────┼───────┤ │ 1 │ The Hobbit │ J.R.R. Tolkien │ 310 │ 1937-09-21 │ 39.99 │ │ 2 │ The Fellowship of the Ring │ J.R.R. Tolkien │ 423 │ 1954-07-29 │ 49.99 │ │ 3 │ The Two Towers │ J.R.R. Tolkien │ 352 │ 1954-11-11 │ 49.99 │ │ 4 │ The Return of the King │ J.R.R. Tolkien │ 416 │ 1955-10-20 │ 49.99 │ └────┴────────────────────────────┴────────────────┴───────┴──────────────┴───────┘ ``` ### CSV The csv mode simply outputs the result in the comma-separated values for columns and newline character for rows. ``` .mode csv SELECT * FROM books; ``` ```csv 1,"The Hobbit","J.R.R. Tolkien",310,1937-09-21,39.99 2,"The Fellowship of the Ring","J.R.R. Tolkien",423,1954-07-29,49.99 3,"The Two Towers","J.R.R. Tolkien",352,1954-11-11,49.99 4,"The Return of the King","J.R.R. Tolkien",416,1955-10-20,49.99 ``` ### Column The column mode simply outputs the result in columns with certain width. We can set the option `.width` to change the width. ``` .mode column SELECT * FROM books; ``` ``` id title author pages release_date price -- -------------------------- -------------- ----- ------------ ----- 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 ``` ### HTML The html output mode simply renders the result set / tables into an table element in html. ``` .mode html SELECT * FROM books; ``` ```html id title author pages release_date price 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 ``` ### Insert The insert mode simply outputs the result in SQL insert statements. This is really handy for bulk inserting or migrating data across databases or tables. ``` .mode insert SELECT * FROM books; ``` ``` INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198); INSERT INTO "table"(id,title,author,pages,release_date,price) VALUES(4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198); ``` ### JSON The json mode simply outputs the result in json format. ``` .mode json SELECT * FROM books; ``` ```json [ {"id":1,"title":"The Hobbit","author":"J.R.R. Tolkien","pages":310,"release_date":"1937-09-21","price":39.99000000000000198}, {"id":2,"title":"The Fellowship of the Ring","author":"J.R.R. Tolkien","pages":423,"release_date":"1954-07-29","price":49.99000000000000198}, {"id":3,"title":"The Two Towers","author":"J.R.R. Tolkien","pages":352,"release_date":"1954-11-11","price":49.99000000000000198}, {"id":4,"title":"The Return of the King","author":"J.R.R. Tolkien","pages":416,"release_date":"1955-10-20","price":49.99000000000000198} ] ``` ### List (The default) As we already know, the list mode is the default mode. ``` .mode list SELECT * FROM books; ``` ``` id|title|author|pages|release_date|price 1|The Hobbit|J.R.R. Tolkien|310|1937-09-21|39.99 2|The Fellowship of the Ring|J.R.R. Tolkien|423|1954-07-29|49.99 3|The Two Towers|J.R.R. Tolkien|352|1954-11-11|49.99 4|The Return of the King|J.R.R. Tolkien|416|1955-10-20|49.99 ``` ### Markdown The markdown mode simply outputs the result in markdown format. We can use this output in a markdown file and it will nicely draw the tables out. ``` .mode markdown SELECT * FROM books; ``` ```md | id | title | author | pages | release_date | price | |----|----------------------------|----------------|-------|--------------|-------| | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | ``` ### Qbox The qbox mode simply outputs the result in box format with the option as `--wrap 60 --quote`. ``` .mode qbox SELECT * FROM books; ``` ``` ┌────┬──────────────────────────────┬──────────────────┬───────┬──────────────┬───────┐ │ id │ title │ author │ pages │ release_date │ price │ ├────┼──────────────────────────────┼──────────────────┼───────┼──────────────┼───────┤ │ 1 │ 'The Hobbit' │ 'J.R.R. Tolkien' │ 310 │ '1937-09-21' │ 39.99 │ │ 2 │ 'The Fellowship of the Ring' │ 'J.R.R. Tolkien' │ 423 │ '1954-07-29' │ 49.99 │ │ 3 │ 'The Two Towers' │ 'J.R.R. Tolkien' │ 352 │ '1954-11-11' │ 49.99 │ │ 4 │ 'The Return of the King' │ 'J.R.R. Tolkien' │ 416 │ '1955-10-20' │ 49.99 │ └────┴──────────────────────────────┴──────────────────┴───────┴──────────────┴───────┘ ``` ### Quote The quote mode simply outputs the result in SQL-string quote format. ``` .mode quote SELECT * FROM books; ``` ``` 'id','title','author','pages','release_date','price' 1,'The Hobbit','J.R.R. Tolkien',310,'1937-09-21',39.99000000000000198 2,'The Fellowship of the Ring','J.R.R. Tolkien',423,'1954-07-29',49.99000000000000198 3,'The Two Towers','J.R.R. Tolkien',352,'1954-11-11',49.99000000000000198 4,'The Return of the King','J.R.R. Tolkien',416,'1955-10-20',49.99000000000000198 ``` ### Table The table mode simply outputs the result in table format. We saw this mode in the previous example. ``` .mode table SELECT * FROM books; ``` ``` +----+----------------------------+----------------+-------+--------------+-------+ | id | title | author | pages | release_date | price | +----+----------------------------+----------------+-------+--------------+-------+ | 1 | The Hobbit | J.R.R. Tolkien | 310 | 1937-09-21 | 39.99 | | 2 | The Fellowship of the Ring | J.R.R. Tolkien | 423 | 1954-07-29 | 49.99 | | 3 | The Two Towers | J.R.R. Tolkien | 352 | 1954-11-11 | 49.99 | | 4 | The Return of the King | J.R.R. Tolkien | 416 | 1955-10-20 | 49.99 | +----+----------------------------+----------------+-------+--------------+-------+ ``` ### Tabs The tabs mode simply outputs the result in tab-separated values. ``` .mode tabs SELECT * FROM books; ``` ``` id title author pages release_date price 1 The Hobbit J.R.R. Tolkien 310 1937-09-21 39.99 2 The Fellowship of the Ring J.R.R. Tolkien 423 1954-07-29 49.99 3 The Two Towers J.R.R. Tolkien 352 1954-11-11 49.99 4 The Return of the King J.R.R. Tolkien 416 1955-10-20 49.99 ``` ### Tcl The Tcl mode simply outputs the result in Tcl format. Tcl or tool command language is a configuration friendly format. ``` .mode tcl SELECT * FROM books; ``` ```tcl "id" "title" "author" "pages" "release_date" "price" "1" "The Hobbit" "J.R.R. Tolkien" "310" "1937-09-21" "39.99" "2" "The Fellowship of the Ring" "J.R.R. Tolkien" "423" "1954-07-29" "49.99" "3" "The Two Towers" "J.R.R. Tolkien" "352" "1954-11-11" "49.99" "4" "The Return of the King" "J.R.R. Tolkien" "416" "1955-10-20" "49.99" ```

SQLite dot commands: change directory command

Sun, 31 Aug 2025 00:00:00 UTC

## Change directory dot command If you are in a sqlite shell and forgot to change directory or want to navigate to a separate directory, you can do that with the `.cd` dot command. ```sqlite .cd /path/to/directory ``` This is better than doing `.shell cd /path/to/directory` because it doesn't open a separate terminal process. So, the `.cd` is persistant throughout the session, whereas the `.shell cd ` will only within that command (subprocess). The `.cd` command changes the working directory of the SQLite shell itself, so the change persists for the rest of the session. This means commands like `.import`, `.read`, or `.output` will automatically look for files in the new directory. However, `.shell cd ` spawns a separate subprocess, and the directory change is discarded as soon as that command finishes. It does not affect SQLite’s own state of the current directory. So if you plan to read or write multiple files from a different location during your SQLite session, prefer the built-in `.cd` command.

SQLite dot commands: run system shell commands

Sat, 30 Aug 2025 00:00:00 UTC

## Shell dot command If you are in middle of a sqlite shell session, and you don't want to quit the shell to run arbitrary shell command, you can simply use the `.shell ` to execute any shell commands right from within the sqlite shell. How handy is this! ```sqlite .shell echo "hello, world!" ``` That is a lame example, but it shows you the power of the `.shell` command. Let's say I want to run a golang project, I can do this: ```sqlite .shell go run main.go ``` Its helpful if you want to do something but you don't want to quit the shell to do that: - look up few files/datapoints from the local filesystem, - run scripts to populate data - populate database and then reopen the db shell This is are the things that I have stumbled upon, so far. Need more experience to see if there are more.

SQLite dot commands: read command is recursive?

Fri, 29 Aug 2025 00:00:00 UTC

Yesterday, while exploring the `.read` command I discovered a little hack. We know you can read arbitrary sqlite shell commands from the `.read | ` operator, even the dot commands. So, you can even use `.read` itself again. Which means we can call the same file to read inside the script. Ops! Let's create a file called `loop.sh` this will emit certain valid sql/sqlite commands like so: ```bash #!/bin/bash echo "SELECT 'hello, world!';" echo ".read |./loop.sh" ``` The first will print out "SELECT 'hello world!';" so when directed to the shell, it will be executed and print "hello, world!" Then the 3rd line `echo ".read |./loop.sh"` is the magic, it will execute the script again right? Because it's a valid sqlite shell command and it will execute it. It will take the read command and print everything when executing the `loop.sh` shell script. What's the output? ```bash $ cat loop.sh #!/bin/bash echo "SELECT 'hello, world!';" echo ".read |./loop.sh" $ sqlite3 SQLite version 3.50.4 2025-07-30 19:33:53 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read |./loop.sh hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! hello, world! Input nesting limit (25) reached at line 2. Check recursion. sqlite> ``` Well, SQLite developers are smarter to handle the edge case I think. :) This is a little fun, but it's kind of a quirk, an not-so easter egg but interesting thing to know.

SQLite dot commands: read file or standard output

Thu, 28 Aug 2025 00:00:00 UTC

## Read dot Command The `.read` dot command is a quick handy command to import and run your SQL queries in the current session. You can just pass a SQL file (usually a query ending with ; it will execute each query one by one) ```bash .read filename ``` Let’s say this is a sql file containing the schema of a database, just one table users. ```sql CREATE TABLE users ( id INTEGER PRIMARY KEY, name TEXT ); ``` Writing this sql query in the `schema.sql` file If we use the .read command with the name of the file it will execute it line by line, (line meaning terminated by ; here or even any dot commands) ```bash .read schema.sql ``` This will just execute the query, if there are SELECT statements possibly then will output the result set too. Let’s add some users to the table with the `insert_users.sql` ```sql INSERT INTO users(name) VALUES('abc'), ('def'), ('ghi'); SELECT * FROM users; ``` Saving this and running .read insert_users.sql should now have displayed all the 3 inserted users. ## Reading with dot commands Let’s create one more file with specific dot commands to render a nice table format of a result set from a select statement ```sql .mode table SELECT * FROM users; ``` I save the above contents into a file called `users.sql` Now if I run the .read command with the users.sql you should see all your users but with the mode as table format. It even takes the dot commands and all of the nice options out there. Be cautious though! ``` .read users.sql ``` ``` sqlite> .read users.sql +----+---------+ | id | name | +----+---------+ | 1 | User_1 | | 2 | User_2 | | 3 | User_3 | | 4 | User_4 | | 5 | User_5 | | 6 | User_6 | | 7 | User_7 | | 8 | User_8 | | 9 | User_9 | | 10 | User_10 | +----+---------+ ``` ## Read with standard output You can also run a script that will generate some SQL commands and execute them. By using the | operator after .read and providing the name of the script. this will run the script and any output of that script (valid sql) will be executed on the shell. For instance I have a script that would insert 10 users with a bash script like so called `inser_users.sh` ```bash #!/bin/bash for i in $(seq 1 10); do echo "INSERT INTO users (name) VALUES ('User_$i');" done ``` If you would use this as is it won’t work as its a bash script, you can’t run those in a sqlite shell. However by using `.read | filename` (shell script), that will indeed do the trick. The important thing to note here is that it would run all the things outputted from the script, so its kind of running the script and the std out of that script is taken as the input for the read command so like typical pipe operator in linux. ``` .read | insert_users.sh ``` This will run the script and the insert statements will be printed, so it will execute them. So, in essence we are basically piping the output of one command to the other in this case the read command. ```sql SELECT * FROM users; ``` As you can see it has now inserted all the 10 users. The `|` command can be followed by any script, not limited to bash, it could be python, or any script or program that can print to the standard output a correct and legal SQL statements. This is a really great small yet highly important piece of functionality of sqlite shell. Reference: - [SQLite CLI: Reading SQL from a File](https://sqlite.org/cli.html#reading_sql_from_a_file)

SQLite Scalar function: 3 valued iif and if scalar function

Wed, 27 Aug 2025 00:00:00 UTC

In the recent SQLite version specifically the 3.48.0, the support for pair of conditions was added to the iif function. And also an alias for iif as if was added for compatibility and ease of use for people coming from other databases. ## IIF Before 3.48 Before this release it was kind of 3 way /ternary operation kind of thing like this ### Simple IF THEN ELSE We can use the IIF function by passing 3 paraeters, first is the condition to check, the second is the value to return if the condition is true and the third is the value to return if the condition is false. ```sql SELECT iif(1=0, 'one is zero', 'one is not zero'); ``` ``` one is not zero ``` ### Ternary like nested IIF Optionally the third condition can be nested with another IIF as it will be a value in the end. So, we can use IIF like this ```sql SELECT iif(7%2=0, 'even', iif(7%3=0, 'multiple of 3', 'prime')) AS is_prime; ``` ``` +---------+ | is_prime| +---------+ | prime | +---------+ ``` That is quite simple but it gets little messed up for more than one check, take this for instance ```sql SELECT iif(2025 % 400 = 0, 'leap', iif(2025 % 100 = 0, 'not leap', iif(2025 % 4 = 0, 'leap', 'not leap'))) AS is_leap_year; ``` ``` +--------------+ | is_leap_year | +--------------+ | not leap | +--------------+ ``` I have printed it in a nice way, but still might get harder to read and even write, notice the number of closing brackets at the end. ## IIF After 3.48 With the latest 3.48 version, this changes quite a bit, you can have iif or if as like a case when then replacement like so: ```sql SELECT iif(2025 % 400 = 0, 'leap', 2025 % 100 = 0, 'not leap', 2025 % 4 = 0, 'leap', 'not leap') AS is_leap_year; ``` ``` +--------------+ | is_leap_year | +--------------+ | not leap | +--------------+ ``` How clean is that, with this you can specify, Condition, Value as a pair and it can have N subsequent pairs of this condition - value. All and this while maintaining backwards compatibility, that’s quite a remarkable thing to push I would say, not easy to handle these. Now it has 3 variants - 3 valued expression (single expression, could be nested too, but three expressions, like → condition, true, false) - 2 valued expression (assumes the false value is NULL) - N condition - valued pair expression (two pair of condition and value) These is neat and makes a lot of sense. Let’s see the same example in all 3 variants ### 3 Valued Expression This is the most simple one ```sql SELECT iif(2025%4=0, 'probably leap', 'not leap') AS is_leap_year; ``` ``` +--------------+ | is_leap_year | +--------------+ | not leap | +--------------+ ``` This returned the false value which happens when the condition evaluated to false. ### 2 Valued Expression ```sql SELECT iif(2025%4=0, 'probably leap') AS is_leap_year; ``` ``` +--------------+ | is_leap_year | +--------------+ | | +--------------+ ``` This returned NULL as 2025 is not divisible by 4. If there are 2N parameters to the function, and if all the conditions are false, it will return NULL. And the above example is the base case of it. Remember to have atleast 2 parameters, one the condition and one as the value if that condition evaluates to true. The last parameter is optional and if the condition is false, it would return NULL. ### N Pair Expression So moving on the final one which is the most readable version of it and allows more stuff ```sql SELECT iif(2025 % 400 = 0, 'leap', 2025 % 100 = 0, 'not leap', 2025 % 4 = 0, 'leap', 'not leap') || ' year' AS is_leap_year; ``` ``` +---------------+ | is_leap_year | +---------------+ | not leap year | +---------------+ ``` Elegant Notice the last condition, if you don’t specify that, it will be NULL, so it would be 2N+1 parameters (where N is the number of conditions) Other way to remember is - Even parameters and all conditions are - True (the first value) or equivalent corresponding value for that condition - False (it will return NULL) - Odd parameters and all conditions are - True (the first value) or equivalent corresponding value for that condition - False (it will return the last parameter which is the false condition) You can use iif or if interchangeably. Both these do the same thing as the CASE WHEN THEN conditions, I think IIF now is really easy to write as now it supports arguments in pairs

SQLite functions: unistr and unistr_quote

Tue, 26 Aug 2025 00:00:00 UTC

SQLite [3.50](https://www.sqlite.org/releaselog/3_50_0.html) introduced [unistr](https://www.sqlite.org/lang_corefunc.html#unistr) and [unistr_quote](https://www.sqlite.org/lang_corefunc.html#unistr_quote) functions These are for taking in escape sequences and returning the actual unicode string like emoji or other non-english characters outside the ascii convention forming the utf-8 character set The `unistr` is used to take in the escape sequences and return the actual unicode string The `unistr_quote` is used to escape those escape sequences in a sql query so your data can be stored with emojis. > I wonder these functions were added because of LLMs emoji-heavy outputs? Here’s how you use unistr in a simple query ```sql SELECT unistr('\u2665, \U0001F600'); ```` This will print the :heart and :smiley emojis I won’t print the emojis, it might look like a AI-generated post (grunts with pride) This is handy to render the values but first you need to get the values stored in the db, for that you need to have it like a string i.e. escaped string That is where the `unistr_quote comes` in ```sql SELECT unistr_quote('\U0001F600'); ``` This will simply print '\U0001F600' notice the quotes here, it will be a escaped character sequence making it like a special string in middle of a string, so if we have something like this ```sql SELECT printf('what the heck is this%s symbol?', unistr_quote('\U0001F449')); ``` > output: what the heck is this'\U0001F449' symbol? This will make the distinction clear, we get a quoted string for that special set of characters or escape sequence which can be useful if the storage medium would make storing emojis obsolete.

SQLite dot commands: html tables with output and once

Mon, 25 Aug 2025 00:00:00 UTC

The `.once -w` or `.www` was released in [SQLite 3.47](https://www.sqlite.org/releaselog/3_47_0.html) release last year. I was trying this command ```shell .once -w results.html ``` But was getting an error that it was not a valid argument, and the .www command doesn’t exist. I saw the git repo and found the source code and found out that it was released in the recent version. My laptop had SQLite 3.37 version installed. I ran a docker image of sqlite to the latest version and check out the working of that .www command or .once -w subcommand option. ## What is .once -w or .output -w or .www All of these are equivalent - `.once -w` - `.www` The output is just for all the subsequent commands to be saved in a file, whereas the `.once -w` and `.www` are only for the next one query. This command basically allows the resultset to be saved in a html format/file. Note that you cannot specify a file, it will be stored in a temporary file, and if that is accesible and readable to the browser and enough permissions are set to the temp folder (whatever that is for linux and windows or mac), it should render an html page (its simple) However, if you are thinking of getting some specific file to write the results then you need to do some work ``` .output result.html .print '' .mode html .headers on SELECT abs(random()%10)+1 as "some numbers", 'number' from generate_series(1,10); .print '

' ``` This will store the html document in the result.html with proper tags and semantics not just the table element Phew! That might not be a lot, but that is the thing, the .mode html will only get you the table element but it wont render properly, browsers won’t be able to read the raw html elemetnts without the html doctype tags.

SQLite: Randomly order rows

Sun, 24 Aug 2025 00:00:00 UTC

Return rows or records in a table with random order in SQLite Let's take a simple table for this: ```sql CREATE TABLE user ( id INTEGER PRIMARY KEY AUTOINCREMENT, name TEXT ); INSERT INTO user (name) VALUES ('abc'), ('def'), ('ghi'), ('jkl'), ('mno'); ``` ```sql SELECT * FROM user ORDER BY random(); ``` This will return the rows in a random order You can limit the number of rows ```sql SELECT * FROM user ORDER BY random() LIMIT 5; ``` We can even randomly shuffle a subset of rows with limit and where clauses ```sql SELECT * FROM user WHERE id BETWEEN 1 and 10 ORDER BY random() limit 5; ``` This is cool if you want to get certain subset of samples but in no strict order.

SQLite Scalar Function: Random

Sat, 23 Aug 2025 00:00:00 UTC

The random function in sqlite is quite handy to generate pseudo random numbers. The random function returns a value between `-9223372036854775807` and `+9223372036854775807` ```sql SELECT CAST(power(2,63) AS INTEGER); SELECT CAST(-power(2,63) AS INTEGER); ``` Note that it doesn't return the max integer as `9223372036854775808` becuase using those might break where integer limit might overflow. This was actually documentated incorrectly before 25th July 2025, that's quite recent. We can use that to get absolute values, that is to avoid negative values and then mod (divide and get the remainder) it with the max number we want to generate upto. Example ```sql SELECT random() as random_number; ``` ``` random_number 7855057830251041076 ``` If we want numbers between specific range then use this ```sql SELECT abs(random() % 10) + 1 as random_number; ``` ``` random_number 6 ``` Here 10 is the max number, 1 is the minimum number in the range, if you want negative as well as positive values, you can remove the abs function ```sql SELECT (random() % 10) + 1 AS random_number FROM generate_series(1,20); ``` This will generae random numbers between - 10 and 10, 20 such numbers. We'll leverage the generate series function that I learned last day.

SQLite functions: Generate_Series Examples

Fri, 22 Aug 2025 00:00:00 UTC

After taking a look at what the Generate Series function does, now let's see some examples that might be handy. ## Simple sequence To generate a series of numbers from 1 to 5. ```sql SELECT * FROM generate_series(1, 5); ``` ## Incremental sequence To generate a series of numbers from 0 to 50 in steps of 10. ```sql SELECT * FROM generate_series(0, 50, 10); ``` This will start from 0, increment by 10 till 50. ``` 0 10 20 30 40 50 ``` ## Backwards sequence To generate a series from 50 to 0 in steps of 10 ```sql SELECT * FROM generate_series(0, 50, -10); ``` This would start from 50 (becuase -ve step will flip the start and stop parameters) and decrement by 10 till 0 ``` 50 40 30 20 10 0 ``` ## Random Numbers To generate 5 random number between 1 to 5. ```sql SELECT * FROM generate_series(1, 5) ORDER BY RANDOM(); ``` This would generate 5 numbers 1 to 5, and the ordering will be random. This is becuase for each row, it will generate a random number between -max int to +max int and then order them based on those numbers. OR To generate 5 random number between 1 and 10. ``` SELECT ABS((random()%10)) + 1 FROM generate_series(1, 5); ``` This would generate 5 random numbers between 1 and 10. The random function generates a integer between -max int to +max int and then mod it to 10 (or the range you want to generate up to) this will leave us with -9 to 9 numbers but it also be negative so we make it ABS which will make it positive, and add one to make the 0 offset to 1. ``` 3 9 10 7 5 ``` ## Random Characters To generate 5 random characters from A to Z. ```sql SELECT char((random()%26)+65) FROM generate_series(1, 5); ``` This would generate a random integer between -max int and + max int, mod it with 26 to get that number between 0 to 25, add 65 which will then can be casted to char in order to render it as an ASCII String. ## Date Ranges To generate dates for a month ```sql SELECT date('2025-08-01', '+' || (value-1) || ' day') as date_val FROM generate_series(1, 31); ``` This will take the start date 2025-08-01 and add the value -1 (the value will be from 1 so we need to omit the first day) as a day. This will then generate a series of dates from 2025-08-01 to 2025-08-31. To generate first day of each month ```sql SELECT date('2025-08-01', '+' || (value-1) || ' month') as date_val FROM generate_series(1, 12); ``` ## Gap filling in existing ids To fill in the missing ids in a sequence of ids ```sql SELECT * FROM generate_series(1, 5) WHERE value NOT IN (1, 3, 5); ``` Here you could imagine the not in list could be another table with the result set only with the ids.

SQLite functions: generate_series

Thu, 21 Aug 2025 00:00:00 UTC

The [generate_series](https://www.sqlite.org/series.html) is a table valued function in sqlite and is available via the [generate_series](https://sqlite.org/src/artifact?ci=trunk&filename=ext/misc/series.c) extension. > The valued function is something that returns a table but is virtual (doesn't really have data or schema in it). It has hidden columns which are used as parameters to the function to constrain the output and generate the data according to those parameters to the function. The basic operation with `generate_series` would look like this: ```sql SELECT * FROM generate_series(1, 5); ``` ``` sqlite> SELECT * FROM generate_series(1, 5); +-------+ | value | +-------+ | 1 | | 2 | | 3 | | 4 | | 5 | +-------+ sqlite> ``` Pretty neat, it generated a series of numbers (integers) from 1 to 5 (5 included). Now are they really integers or? SQLite is wired, it doesn't really have types, but they are integers as default so to confirm: ```sql SELECT typeof(value) FROM generate_series(1, 5); ``` ``` sqlite> SELECT typeof(value) FROM generate_series(1, 5); +---------------+ | typeof(value) | +---------------+ | integer | | integer | | integer | | integer | | integer | +---------------+ ``` OK! Cool, now let's take this to a next level ```sql SELECT char(value+64) FROM generate_series(1, 5); ``` ``` sqlite> SELECT char(value+64) FROM generate_series(1, 5); +----------------+ | char(value+64) | +----------------+ | A | | B | | C | | D | | E | +----------------+ ``` OK! That is simple ascii and type casting shenanigans! We cast the value which would be integer 1 to 5 and add 64 to it to make it in the range 65 to 70 which would be ASCII equivalent of A to E, you could do the same thing but different parameters: ``` SELECT char(value) FROM generate_series(65, 70); ``` ``` sqlite> SELECT char(value) FROM generate_series(65, 70); +-------------+ | char(value) | +-------------+ | A | | B | | C | | D | | E | | F | +-------------+ ``` Same result! But, ok, this is other rabbit hole, let's focus back on generate_series, the parameters are start and stop right? Well there is one more, step. This will be like the incremental number for each step. If you are familiar with C styled for loops, you know this as the increment part, or the `i++` bit. ```sql SELECT value FROM generate_series(0, 20, 5) ``` ``` sqlite> SELECT value FROM generate_series(0, 20, 5); +-------+ | value | +-------+ | 0 | | 5 | | 10 | | 15 | | 20 | +-------+ sqlite> ``` So, the third parameter increments the counter by that number. So we start from 0 and increment 5 for each step (row) till 20. Default step value is 1, however the quirk is 0 is still considered as 1. You can also go reverse ```sql SELECT * FROM generate_series(0, 20, -5); ``` This will generate number series as 20,15,10,5,0: Note that the step is -1 not the stop value, this will basically flip the start and stop values, so we will start from 20 and decrement by 5 at each step. If you want to generate a series in negative ranges, the start or stop needs to modified accordingly, the start and stop control which numbers you iterate and the step controls in which direction it moves from start to end. ```sql SELECT * FROM generate_series(-20, 20, -5); ``` This will generate the number series from 20 to -20, start from 20 and decrement by 5 each step until we reach -20. So we had a start as -20 but why do we start from 20? well because the step is negative, we need to start from the higher value and end at the lower value, hence the start and stop will be flipped for a step value < 0. ```sql SELECT * FROM generate_series(10) LIMIT 5; ``` This gave 10,11,12,13,14 right, how? Because there is not just one column in the generate_series virtual table there are more Apart from value that is the integer we get from the function as the row (column cell) there are start, stop, step and rowid columns which are hidden columns in the `generate_series` function or virtual table Let’s see SELECT rowid, start, stop, step, value FROM generate_series(0,10,2); ``` +-------+-------+------+------+-------+ | rowid | start | stop | step | value | +-------+-------+------+------+-------+ | 1 | 0 | 10 | 2 | 0 | | 2 | 0 | 10 | 2 | 2 | | 3 | 0 | 10 | 2 | 4 | | 4 | 0 | 10 | 2 | 6 | | 5 | 0 | 10 | 2 | 8 | | 6 | 0 | 10 | 2 | 10 | +-------+-------+------+------+-------+ ``` The start, stop, step values remain same, as those are the parameters those can’t change but notice the rowid and the value the first and the last column in the result set do change. Interestingly the start, stop, and step are the parameters to the function, so you can technically pass them however you like ```sql SELECT * FROM generate_series() WHERE start = 10 AND stop = 20 AND step = 5; ``` This will give 10,15,20 since the start is 10, increment by 5 until 20. The step is optional, note that stop is optional too, but never forget to provide it, it might cause a forever loop and keep on incrementing by 1 and never give the result back until your computer crashes.

SQLite Dot command: Once

Wed, 20 Aug 2025 00:00:00 UTC

The [once](https://sqlite.org/cli.html#writing_results_to_a_file) dot command is similar to the [output](https://sqlite.org/cli.html#writing_results_to_a_file), however the distinction is that it is limited to the very next SQL query and not all the subsequent queries. The primary way to use `once` is either by specifying the specific file name/path or opening the result set in the system editor. ### Output to a file ``` .once somefile.txt ``` Once this is set, any query you execute, the result set of it will be logged to the specified file. ```sql SELECT printf("Hello %s! Bye, %s!", "world", "mars"); ``` ``` $ cat somefile.txt Hello world! Bye, mars! ``` Pretty cool! But there's more There are three options available to perform different things for where this output can go - Into a temporary file opening in a system text editor with -e - Into a temporary excel/csv/xlsx file in a system spreadsheet editor application with the -x option - Into an excel/csv/xlsx file (compatible with Microsoft Excel) containing utf-8 character or symbols with the --bom option. ### Text Editor You can open the result set to a temporary file in a system text editor with the -e option. ``` .once -e ``` You can set the system editor with ### Spreadsheet Editing Application Also you can open the result set into a temporary xlsx or csv file in a system spreadsheet editing application like Microsoft Excel, LibreOffice Calc, etc. ``` .once -x ``` ### Unicode encoded Spreadsheet This option is used to create an excel/csv/xlsx file (compatible with Microsoft Excel) containing utf-8 character or symbols. If you used -x on Linux, it would work fine there, but the file format won't be compatible in the Microsoft Excel or other xlsx formats. To make sure the utf-8 characters are rendered and parsed properly in the excel file, just use the --bom which stands for byte order mark. This options adds certain bytes at the beginning of the file to make the application understand which encoding to use while rendering to use, like `EF BB BF` for utf-8, `FE FF` for utf-16, and so on. ``` .mode csv .once --bom filename.xlsx ```

SQLite Dot command: Output

Tue, 19 Aug 2025 00:00:00 UTC

I will be starting to post something I learn daily about sql,sqlite or anything sql. There is so much to learn! Today I found you can output a result of query in sqlite shell with ``` .output filename ``` This will start appending the result of all the queries executed in the shell to the file. It will preserve all the modes and configuration used for that specific instance of the shell. Basically just the output you would see after executing the query n the shell, it will dumo that to the mentioned file (it will not output to the screen/io as it will dump in the file) More info here: [SQLite Dot command: Output](https://sqlite.org/cli.html#writing_results_to_a_file)