https://www.meetgor.com Posts tagged with advent-of-sql en-us Sat, 18 Apr 2026 05:00:11 UTC https://www.meetgor.com/sqlog/advent-of-sql-2025-day-15 Advent of SQL, Day 15 - Confirmation Phrase Dispatches We are on the final day of Advent of SQL! I can't believe it, I completed it! (with some help of course, Sun, 28 Dec 2025 00:00:00 UTC ## Advent of SQL, Day 15 - Confirmation Phrase Dispatches We are on the final day of Advent of SQL! I can't believe it, I completed it! (with some help of course, I ~~don't~~ didn't know SQL very well. But this 15 days flipped it around.). I feel good and fresh to start the year to go deep in SQLite and databases! Let's solve the last day first, if it has somethings reamining to teach us! We need to make a few changes. SQLite doesn't have `->>` operator for extracting JSON values from the columns, so we need to use `json_extract` instead. So, replace this 9th line from and with: ```diff - marker_letter TEXT GENERATED ALWAYS AS (payload ->> 'marker') STORED, + marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, ``` Once we have it, we also need to remove the marker `::jsonb` from the each insert row, we can do that with `sed` or any other utility you like. ``` sed "s/'\(.*\)'::jsonb/'\1'/g" day15-inserts.sql > day15-inserts-sqlite.sql ``` ```sql DROP TABLE IF EXISTS incoming_dispatches; DROP TABLE IF EXISTS system_dispatches; CREATE TABLE system_dispatches ( id SERIAL PRIMARY KEY, system_id TEXT NOT NULL, dispatched_at TIMESTAMP NOT NULL, payload JSONB NOT NULL, marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, UNIQUE (system_id, dispatched_at, payload) ); CREATE TABLE incoming_dispatches ( system_id TEXT, dispatched_at TIMESTAMP, payload JSONB ); INSERT INTO system_dispatches (system_id, dispatched_at, payload) VALUES ('SYS-0081', '2025-12-21T06:02:26', '{"marker": "T", "source": "secondary"}'), ('SYS-0137', '2025-12-19T06:03:21', '{"marker": "D", "source": "secondary"}'), ('SYS-0237', '2025-12-19T06:23:37', '{"marker": "D", "source": "secondary"}'), ('SYS-0006', '2025-12-24T18:10:16', '{"marker": "K", "source": "secondary"}'), ('SYS-0170', '2025-12-19T06:17:24', '{"marker": "I", "source": "secondary"}'), ('SYS-0224', '2025-12-19T06:23:24', '{"marker": "S", "source": "secondary"}'), ('SYS-0007', '2025-12-24T18:10:06', '{"marker": "O", "source": "secondary"}'), ('SYS-0035', '2025-12-23T15:55:34', '{"marker": "C", "source": "secondary"}'), ('SYS-0010', '2025-12-23T15:55:09', '{"marker": "L", "source": "secondary"}'), ('SYS-0037', '2025-12-23T15:55:36', '{"marker": "D", "source": "secondary"}'), ('SYS-0001', '2025-12-24T08:02:00', '{"marker": "X", "source": "primary"}'), ('SYS-0225', '2025-12-19T06:23:25', '{"marker": "Q", "source": "secondary"}'), ('SYS-0095', '2025-12-21T06:02:40', '{"marker": "D", "source": "secondary"}'), ('SYS-0021', '2025-12-23T15:55:20', '{"marker": "W", "source": "secondary"}'), ('SYS-0001', '2025-12-24T08:10:00', '{"marker": "A", "source": "primary"}'), ('SYS-0009', '2025-12-24T08:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0119', '2025-12-19T06:03:03', '{"marker": "E", "source": "secondary"}'), ('SYS-0142', '2025-12-19T06:03:26', '{"marker": "O", "source": "secondary"}'), ('SYS-0048', '2025-12-23T15:55:30', '{"marker": "I", "source": "secondary"}'), ('SYS-0228', '2025-12-19T06:23:28', '{"marker": "G", "source": "secondary"}'), ('SYS-0092', '2025-12-21T06:02:37', '{"marker": "K", "source": "secondary"}'), ('SYS-0008', '2025-12-24T08:02:07', '{"marker": "B", "source": "primary"}'), ('SYS-0111', '2025-12-19T06:02:55', '{"marker": "W", "source": "secondary"}'), ('SYS-0106', '2025-12-19T06:02:50', '{"marker": "L", "source": "secondary"}'), ('SYS-0082', '2025-12-21T06:02:27', '{"marker": "O", "source": "secondary"}'), ('SYS-0180', '2025-12-19T06:17:31', '{"marker": "N", "source": "secondary"}'), ('SYS-0103', '2025-12-19T06:02:47', '{"marker": "F", "source": "secondary"}'), ('SYS-0096', '2025-12-21T06:02:41', '{"marker": "A", "source": "secondary"}'), ('SYS-0003', '2025-12-23T06:02:02', '{"marker": "Z", "source": "primary"}'), ('SYS-0157', '2025-12-19T06:17:11', '{"marker": "D", "source": "secondary"}'), ('SYS-0010', '2025-12-23T06:02:09', '{"marker": "C", "source": "primary"}'), ('SYS-0102', '2025-12-19T06:02:46', '{"marker": "O", "source": "secondary"}'), ('SYS-0072', '2025-12-23T06:02:18', '{"marker": "K", "source": "secondary"}'), ('SYS-0175', '2025-12-19T06:17:29', '{"marker": "D", "source": "secondary"}'), ('SYS-0214', '2025-12-19T06:23:14', '{"marker": "U", "source": "secondary"}'), ('SYS-0005', '2025-12-24T08:02:04', '{"marker": "P", "source": "primary"}'), ('SYS-0216', '2025-12-19T06:23:16', '{"marker": "A", "source": "secondary"}'), ('SYS-0047', '2025-12-23T15:55:46', '{"marker": "H", "source": "secondary"}'), ('SYS-0019', '2025-12-23T15:55:18', '{"marker": "H", "source": "secondary"}'), ('SYS-0043', '2025-12-23T15:55:42', '{"marker": "Q", "source": "secondary"}'), ('SYS-0039', '2025-12-23T15:55:38', '{"marker": "L", "source": "secondary"}'), ('SYS-0212', '2025-12-19T06:23:12', '{"marker": "K", "source": "secondary"}'), ('SYS-0133', '2025-12-19T06:03:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0005', '2025-12-24T08:10:04', '{"marker": "N", "source": "primary"}'), ('SYS-0011', '2025-12-24T08:02:10', '{"marker": "J", "source": "primary"}'), ('SYS-0080', '2025-12-21T06:02:25', '{"marker": "N", "source": "secondary"}'), ('SYS-0230', '2025-12-19T06:23:30', '{"marker": "I", "source": "secondary"}'), ('SYS-0100', '2025-12-21T06:02:45', '{"marker": "N", "source": "secondary"}'), ('SYS-0061', '2025-12-23T06:02:07', '{"marker": "T", "source": "secondary"}'), ('SYS-0022', '2025-12-23T15:55:21', '{"marker": "K", "source": "secondary"}'), ('SYS-0149', '2025-12-19T06:17:03', '{"marker": "H", "source": "secondary"}'), ('SYS-0009', '2025-12-23T15:55:08', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-24T08:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0132', '2025-12-19T06:03:16', '{"marker": "K", "source": "secondary"}'), ('SYS-0033', '2025-12-23T15:55:32', '{"marker": "W", "source": "secondary"}'), ('SYS-0193', '2025-12-19T06:17:44', '{"marker": "C", "source": "secondary"}'), ('SYS-0167', '2025-12-19T06:17:21', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-24T08:10:07', '{"marker": "F", "source": "primary"}'), ('SYS-0186', '2025-12-19T06:17:37', '{"marker": "L", "source": "secondary"}'), ('SYS-0078', '2025-12-21T06:02:23', '{"marker": "V", "source": "secondary"}'), ('SYS-0085', '2025-12-21T06:02:30', '{"marker": "Q", "source": "secondary"}'), ('SYS-0046', '2025-12-23T15:55:45', '{"marker": "G", "source": "secondary"}'), ('SYS-0054', '2025-12-23T15:55:36', '{"marker": "W", "source": "secondary"}'), ('SYS-0235', '2025-12-19T06:23:35', '{"marker": "D", "source": "secondary"}'), ('SYS-0089', '2025-12-21T06:02:34', '{"marker": "H", "source": "secondary"}'), ('SYS-0011', '2025-12-23T06:02:10', '{"marker": "J", "source": "primary"}'), ('SYS-0034', '2025-12-23T15:55:33', '{"marker": "K", "source": "secondary"}'), ('SYS-0129', '2025-12-19T06:03:13', '{"marker": "H", "source": "secondary"}'), ('SYS-0211', '2025-12-19T06:23:11', '{"marker": "W", "source": "secondary"}'), ('SYS-0151', '2025-12-19T06:17:05', '{"marker": "W", "source": "secondary"}'), ('SYS-0204', '2025-12-19T06:23:04', '{"marker": "S", "source": "secondary"}'), ('SYS-0213', '2025-12-19T06:23:13', '{"marker": "C", "source": "secondary"}'), ('SYS-0147', '2025-12-19T06:03:31', '{"marker": "A", "source": "secondary"}'), ('SYS-0140', '2025-12-19T06:03:24', '{"marker": "N", "source": "secondary"}'), ('SYS-0123', '2025-12-19T06:03:07', '{"marker": "F", "source": "secondary"}'), ('SYS-0052', '2025-12-23T15:55:34', '{"marker": "H", "source": "secondary"}'), ('SYS-0148', '2025-12-19T06:17:02', '{"marker": "G", "source": "secondary"}'), ('SYS-0059', '2025-12-23T06:02:05', '{"marker": "E", "source": "secondary"}'), ('SYS-0027', '2025-12-23T15:55:26', '{"marker": "K", "source": "secondary"}'), ('SYS-0055', '2025-12-23T15:55:37', '{"marker": "K", "source": "secondary"}'), ('SYS-0155', '2025-12-19T06:17:09', '{"marker": "D", "source": "secondary"}'), ('SYS-0107', '2025-12-19T06:02:51', '{"marker": "A", "source": "secondary"}'), ('SYS-0173', '2025-12-19T06:17:27', '{"marker": "C", "source": "secondary"}'), ('SYS-0007', '2025-12-23T06:02:06', '{"marker": "K", "source": "primary"}'), ('SYS-0016', '2025-12-23T15:55:15', '{"marker": "L", "source": "secondary"}'), ('SYS-0168', '2025-12-19T06:17:22', '{"marker": "G", "source": "secondary"}'), ('SYS-0196', '2025-12-19T06:17:47', '{"marker": "A", "source": "secondary"}'), ('SYS-0234', '2025-12-19T06:23:34', '{"marker": "U", "source": "secondary"}'), ('SYS-0210', '2025-12-19T06:23:10', '{"marker": "I", "source": "secondary"}'), ('SYS-0135', '2025-12-19T06:03:19', '{"marker": "D", "source": "secondary"}'), ('SYS-0150', '2025-12-19T06:17:04', '{"marker": "I", "source": "secondary"}'), ('SYS-0091', '2025-12-21T06:02:36', '{"marker": "W", "source": "secondary"}'), ('SYS-0154', '2025-12-19T06:17:08', '{"marker": "U", "source": "secondary"}'), ('SYS-0223', '2025-12-19T06:23:23', '{"marker": "F", "source": "secondary"}'), ('SYS-0020', '2025-12-23T15:55:19', '{"marker": "I", "source": "secondary"}'), ('SYS-0160', '2025-12-19T06:17:14', '{"marker": "N", "source": "secondary"}'), ('SYS-0128', '2025-12-19T06:03:12', '{"marker": "G", "source": "secondary"}'), ('SYS-0194', '2025-12-19T06:17:45', '{"marker": "U", "source": "secondary"}'), ('SYS-0231', '2025-12-19T06:23:31', '{"marker": "W", "source": "secondary"}'), ('SYS-0045', '2025-12-23T15:55:44', '{"marker": "A", "source": "secondary"}'), ('SYS-0190', '2025-12-19T06:17:41', '{"marker": "I", "source": "secondary"}'), ('SYS-0197', '2025-12-19T06:17:48', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-23T06:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0008', '2025-12-24T18:10:07', '{"marker": "F", "source": "secondary"}'), ('SYS-0158', '2025-12-19T06:17:12', '{"marker": "V", "source": "secondary"}'), ('SYS-0003', '2025-12-24T08:10:02', '{"marker": "V", "source": "primary"}'), ('SYS-0171', '2025-12-19T06:17:25', '{"marker": "W", "source": "secondary"}'), ('SYS-0229', '2025-12-19T06:23:29', '{"marker": "H", "source": "secondary"}'), ('SYS-0079', '2025-12-21T06:02:24', '{"marker": "E", "source": "secondary"}'), ('SYS-0215', '2025-12-19T06:23:15', '{"marker": "D", "source": "secondary"}'), ('SYS-0233', '2025-12-19T06:23:33', '{"marker": "C", "source": "secondary"}'), ('SYS-0060', '2025-12-23T06:02:06', '{"marker": "N", "source": "secondary"}'), ('SYS-0026', '2025-12-23T15:55:25', '{"marker": "W", "source": "secondary"}'), ('SYS-0125', '2025-12-19T06:03:09', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-24T18:10:00', '{"marker": "A", "source": "secondary"}'), ('SYS-0136', '2025-12-19T06:03:20', '{"marker": "A", "source": "secondary"}'), ('SYS-0010', '2025-12-24T08:02:09', '{"marker": "C", "source": "primary"}'), ('SYS-0010', '2025-12-24T08:10:09', '{"marker": "Q", "source": "primary"}'), ('SYS-0044', '2025-12-23T15:55:43', '{"marker": "L", "source": "secondary"}'), ('SYS-0115', '2025-12-19T06:02:59', '{"marker": "D", "source": "secondary"}'), ('SYS-0005', '2025-12-24T18:10:15', '{"marker": "W", "source": "secondary"}'), ('SYS-0094', '2025-12-21T06:02:39', '{"marker": "U", "source": "secondary"}'), ('SYS-0191', '2025-12-19T06:17:42', '{"marker": "W", "source": "secondary"}'), ('SYS-0182', '2025-12-19T06:17:33', '{"marker": "O", "source": "secondary"}'), ('SYS-0195', '2025-12-19T06:17:46', '{"marker": "D", "source": "secondary"}'), ('SYS-0227', '2025-12-19T06:23:27', '{"marker": "A", "source": "secondary"}'), ('SYS-0032', '2025-12-23T15:55:31', '{"marker": "I", "source": "secondary"}'), ('SYS-0177', '2025-12-19T06:17:31', '{"marker": "D", "source": "secondary"}'), ('SYS-0068', '2025-12-23T06:02:14', '{"marker": "G", "source": "secondary"}'), ('SYS-0069', '2025-12-23T06:02:15', '{"marker": "H", "source": "secondary"}'), ('SYS-0009', '2025-12-23T06:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0176', '2025-12-19T06:17:30', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-23T06:02:07', '{"marker": "B", "source": "primary"}'), ('SYS-0099', '2025-12-21T06:02:44', '{"marker": "E", "source": "secondary"}'), ('SYS-0014', '2025-12-23T15:55:13', '{"marker": "S", "source": "secondary"}'), ('SYS-0042', '2025-12-23T15:55:41', '{"marker": "S", "source": "secondary"}'), ('SYS-0006', '2025-12-24T08:02:05', '{"marker": "R", "source": "primary"}'), ('SYS-0187', '2025-12-19T06:17:38', '{"marker": "A", "source": "secondary"}'), ('SYS-0174', '2025-12-19T06:17:28', '{"marker": "U", "source": "secondary"}'), ('SYS-0056', '2025-12-23T15:55:55', '{"marker": "A", "source": "secondary"}'), ('SYS-0199', '2025-12-19T06:17:50', '{"marker": "E", "source": "secondary"}'), ('SYS-0120', '2025-12-19T06:03:04', '{"marker": "N", "source": "secondary"}'), ('SYS-0003', '2025-12-24T08:02:02', '{"marker": "Z", "source": "primary"}'), ('SYS-0116', '2025-12-19T06:02:59', '{"marker": "A", "source": "secondary"}'), ('SYS-0008', '2025-12-24T18:10:18', '{"marker": "U", "source": "secondary"}'), ('SYS-0205', '2025-12-19T06:23:05', '{"marker": "Q", "source": "secondary"}'), ('SYS-0006', '2025-12-24T08:10:05', '{"marker": "T", "source": "primary"}'), ('SYS-0145', '2025-12-19T06:03:29', '{"marker": "Q", "source": "secondary"}'), ('SYS-0004', '2025-12-24T08:10:03', '{"marker": "E", "source": "primary"}'), ('SYS-0065', '2025-12-23T06:02:11', '{"marker": "Q", "source": "secondary"}'), ('SYS-0086', '2025-12-21T06:02:31', '{"marker": "L", "source": "secondary"}'), ('SYS-0188', '2025-12-19T06:17:39', '{"marker": "G", "source": "secondary"}'), ('SYS-0183', '2025-12-19T06:17:34', '{"marker": "F", "source": "secondary"}'), ('SYS-0062', '2025-12-23T06:02:08', '{"marker": "O", "source": "secondary"}'), ('SYS-0058', '2025-12-23T15:55:57', '{"marker": "V", "source": "secondary"}'), ('SYS-0169', '2025-12-19T06:17:23', '{"marker": "H", "source": "secondary"}'), ('SYS-0105', '2025-12-19T06:02:49', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-24T18:10:11', '{"marker": "A", "source": "secondary"}'), ('SYS-0005', '2025-12-23T06:02:04', '{"marker": "P", "source": "primary"}'), ('SYS-0131', '2025-12-19T06:03:15', '{"marker": "W", "source": "secondary"}'), ('SYS-0232', '2025-12-19T06:23:32', '{"marker": "K", "source": "secondary"}'), ('SYS-0028', '2025-12-23T15:55:27', '{"marker": "L", "source": "secondary"}'), ('SYS-0070', '2025-12-23T06:02:16', '{"marker": "I", "source": "secondary"}'), ('SYS-0029', '2025-12-23T15:55:28', '{"marker": "A", "source": "secondary"}'), ('SYS-0127', '2025-12-19T06:03:11', '{"marker": "A", "source": "secondary"}'), ('SYS-0010', '2025-12-24T18:10:09', '{"marker": "Q", "source": "secondary"}'), ('SYS-0025', '2025-12-23T15:55:24', '{"marker": "I", "source": "secondary"}'), ('SYS-0164', '2025-12-19T06:17:18', '{"marker": "S", "source": "secondary"}'), ('SYS-0114', '2025-12-19T06:02:58', '{"marker": "U", "source": "secondary"}'), ('SYS-0009', '2025-12-24T18:10:08', '{"marker": "S", "source": "secondary"}'), ('SYS-0184', '2025-12-19T06:17:35', '{"marker": "S", "source": "secondary"}'), ('SYS-0003', '2025-12-24T18:10:02', '{"marker": "V", "source": "secondary"}'), ('SYS-0053', '2025-12-23T15:55:35', '{"marker": "I", "source": "secondary"}'), ('SYS-0117', '2025-12-19T06:03:01', '{"marker": "D", "source": "secondary"}'), ('SYS-0074', '2025-12-23T06:02:20', '{"marker": "U", "source": "secondary"}'), ('SYS-0198', '2025-12-19T06:17:49', '{"marker": "V", "source": "secondary"}'), ('SYS-0036', '2025-12-23T15:55:35', '{"marker": "U", "source": "secondary"}'), ('SYS-0112', '2025-12-19T06:02:56', '{"marker": "K", "source": "secondary"}'), ('SYS-0023', '2025-12-23T15:55:22', '{"marker": "C", "source": "secondary"}'), ('SYS-0121', '2025-12-19T06:03:05', '{"marker": "T", "source": "secondary"}'), ('SYS-0181', '2025-12-19T06:17:32', '{"marker": "T", "source": "secondary"}'), ('SYS-0003', '2025-12-24T18:10:13', '{"marker": "H", "source": "secondary"}'), ('SYS-0130', '2025-12-19T06:03:14', '{"marker": "I", "source": "secondary"}'), ('SYS-0066', '2025-12-23T06:02:12', '{"marker": "L", "source": "secondary"}'), ('SYS-0203', '2025-12-19T06:17:54', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-24T18:10:01', '{"marker": "D", "source": "secondary"}'), ('SYS-0104', '2025-12-19T06:02:48', '{"marker": "S", "source": "secondary"}'), ('SYS-0017', '2025-12-23T15:55:16', '{"marker": "A", "source": "secondary"}'), ('SYS-0004', '2025-12-24T18:10:03', '{"marker": "E", "source": "secondary"}'), ('SYS-0238', '2025-12-19T06:23:38', '{"marker": "D", "source": "secondary"}'), ('SYS-0110', '2025-12-19T06:02:54', '{"marker": "I", "source": "secondary"}'), ('SYS-0071', '2025-12-23T06:02:17', '{"marker": "W", "source": "secondary"}'), ('SYS-0004', '2025-12-24T18:10:14', '{"marker": "I", "source": "secondary"}'), ('SYS-0031', '2025-12-23T15:55:30', '{"marker": "H", "source": "secondary"}'), ('SYS-0004', '2025-12-23T06:02:03', '{"marker": "M", "source": "primary"}'), ('SYS-0141', '2025-12-19T06:03:25', '{"marker": "T", "source": "secondary"}'), ('SYS-0207', '2025-12-19T06:23:07', '{"marker": "A", "source": "secondary"}'), ('SYS-0122', '2025-12-19T06:03:06', '{"marker": "O", "source": "secondary"}'), ('SYS-0006', '2025-12-24T18:10:05', '{"marker": "T", "source": "secondary"}'), ('SYS-0030', '2025-12-23T15:55:29', '{"marker": "G", "source": "secondary"}'), ('SYS-0192', '2025-12-19T06:17:43', '{"marker": "K", "source": "secondary"}'), ('SYS-0144', '2025-12-19T06:03:28', '{"marker": "S", "source": "secondary"}'), ('SYS-0098', '2025-12-21T06:02:43', '{"marker": "V", "source": "secondary"}'), ('SYS-0200', '2025-12-19T06:17:51', '{"marker": "N", "source": "secondary"}'), ('SYS-0007', '2025-12-24T08:02:06', '{"marker": "K", "source": "primary"}'), ('SYS-0220', '2025-12-19T06:23:20', '{"marker": "N", "source": "secondary"}'), ('SYS-0063', '2025-12-23T06:02:09', '{"marker": "F", "source": "secondary"}'), ('SYS-0109', '2025-12-19T06:02:53', '{"marker": "H", "source": "secondary"}'), ('SYS-0179', '2025-12-19T06:17:30', '{"marker": "E", "source": "secondary"}'), ('SYS-0163', '2025-12-19T06:17:17', '{"marker": "F", "source": "secondary"}'), ('SYS-0162', '2025-12-19T06:17:16', '{"marker": "O", "source": "secondary"}'), ('SYS-0222', '2025-12-19T06:23:22', '{"marker": "O", "source": "secondary"}'), ('SYS-0134', '2025-12-19T06:03:18', '{"marker": "U", "source": "secondary"}'), ('SYS-0004', '2025-12-24T08:02:03', '{"marker": "M", "source": "primary"}'), ('SYS-0139', '2025-12-19T06:03:23', '{"marker": "E", "source": "secondary"}'), ('SYS-0097', '2025-12-21T06:02:42', '{"marker": "D", "source": "secondary"}'), ('SYS-0146', '2025-12-19T06:03:30', '{"marker": "L", "source": "secondary"}'), ('SYS-0101', '2025-12-19T06:02:45', '{"marker": "T", "source": "secondary"}'), ('SYS-0201', '2025-12-19T06:17:52', '{"marker": "T", "source": "secondary"}'), ('SYS-0051', '2025-12-23T15:55:33', '{"marker": "C", "source": "secondary"}'), ('SYS-0172', '2025-12-19T06:17:26', '{"marker": "K", "source": "secondary"}'), ('SYS-0083', '2025-12-21T06:02:28', '{"marker": "F", "source": "secondary"}'), ('SYS-0009', '2025-12-24T08:10:08', '{"marker": "S", "source": "primary"}'), ('SYS-0124', '2025-12-19T06:03:08', '{"marker": "S", "source": "secondary"}'), ('SYS-0041', '2025-12-23T15:55:40', '{"marker": "F", "source": "secondary"}'), ('SYS-0049', '2025-12-23T15:55:31', '{"marker": "W", "source": "secondary"}'), ('SYS-0202', '2025-12-19T06:17:53', '{"marker": "O", "source": "secondary"}'), ('SYS-0178', '2025-12-19T06:17:29', '{"marker": "V", "source": "secondary"}'), ('SYS-0090', '2025-12-21T06:02:35', '{"marker": "I", "source": "secondary"}'), ('SYS-0006', '2025-12-23T06:02:05', '{"marker": "R", "source": "primary"}'), ('SYS-0073', '2025-12-23T06:02:19', '{"marker": "C", "source": "secondary"}'), ('SYS-0076', '2025-12-23T06:02:22', '{"marker": "A", "source": "secondary"}'), ('SYS-0217', '2025-12-19T06:23:17', '{"marker": "D", "source": "secondary"}'), ('SYS-0159', '2025-12-19T06:17:13', '{"marker": "E", "source": "secondary"}'), ('SYS-0161', '2025-12-19T06:17:15', '{"marker": "T", "source": "secondary"}'), ('SYS-0221', '2025-12-19T06:23:21', '{"marker": "T", "source": "secondary"}'), ('SYS-0236', '2025-12-19T06:23:36', '{"marker": "U", "source": "secondary"}'), ('SYS-0064', '2025-12-23T06:02:10', '{"marker": "S", "source": "secondary"}'), ('SYS-0126', '2025-12-19T06:03:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0024', '2025-12-23T15:55:23', '{"marker": "H", "source": "secondary"}'), ('SYS-0113', '2025-12-19T06:02:57', '{"marker": "C", "source": "secondary"}'), ('SYS-0038', '2025-12-23T15:55:37', '{"marker": "L", "source": "secondary"}'), ('SYS-0075', '2025-12-23T06:02:21', '{"marker": "D", "source": "secondary"}'), ('SYS-0088', '2025-12-21T06:02:33', '{"marker": "G", "source": "secondary"}'), ('SYS-0007', '2025-12-24T08:10:06', '{"marker": "O", "source": "primary"}'), ('SYS-0067', '2025-12-23T06:02:13', '{"marker": "A", "source": "secondary"}'), ('SYS-0156', '2025-12-19T06:17:10', '{"marker": "A", "source": "secondary"}'), ('SYS-0040', '2025-12-23T15:55:39', '{"marker": "O", "source": "secondary"}'), ('SYS-0219', '2025-12-19T06:23:19', '{"marker": "E", "source": "secondary"}'), ('SYS-0166', '2025-12-19T06:17:20', '{"marker": "L", "source": "secondary"}'), ('SYS-0007', '2025-12-24T18:10:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0153', '2025-12-19T06:17:07', '{"marker": "C", "source": "secondary"}'), ('SYS-0077', '2025-12-23T06:02:23', '{"marker": "D", "source": "secondary"}'), ('SYS-0084', '2025-12-21T06:02:29', '{"marker": "S", "source": "secondary"}'); INSERT INTO incoming_dispatches (system_id, dispatched_at, payload) VALUES ('SYS-0013', '2025-12-23T15:55:12', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-24T08:10:01', '{"marker": "D", "source": "primary"}'), ('SYS-0012', '2025-12-23T15:55:11', '{"marker": "O", "source": "secondary"}'), ('SYS-0108', '2025-12-19T06:02:52', '{"marker": "G", "source": "secondary"}'), ('SYS-0165', '2025-12-19T06:17:19', '{"marker": "Q", "source": "secondary"}'), ('SYS-0189', '2025-12-19T06:17:40', '{"marker": "H", "source": "secondary"}'), ('SYS-0050', '2025-12-23T15:55:32', '{"marker": "K", "source": "secondary"}'), ('SYS-0138', '2025-12-19T06:03:22', '{"marker": "V", "source": "secondary"}'), ('SYS-0087', '2025-12-21T06:02:32', '{"marker": "A", "source": "secondary"}'), ('SYS-0208', '2025-12-19T06:23:08', '{"marker": "G", "source": "secondary"}'), ('SYS-0057', '2025-12-23T15:55:56', '{"marker": "D", "source": "secondary"}'), ('SYS-0002', '2025-12-24T18:10:12', '{"marker": "G", "source": "secondary"}'), ('SYS-0011', '2025-12-24T08:10:10', '{"marker": "L", "source": "primary"}'), ('SYS-0226', '2025-12-19T06:23:26', '{"marker": "L", "source": "secondary"}'), ('SYS-0206', '2025-12-19T06:23:06', '{"marker": "L", "source": "secondary"}'), ('SYS-0118', '2025-12-19T06:03:02', '{"marker": "V", "source": "secondary"}'), ('SYS-0185', '2025-12-19T06:17:36', '{"marker": "Q", "source": "secondary"}'), ('SYS-0218', '2025-12-19T06:23:18', '{"marker": "V", "source": "secondary"}'), ('SYS-0018', '2025-12-23T15:55:17', '{"marker": "G", "source": "secondary"}'), ('SYS-0005', '2025-12-24T18:10:04', '{"marker": "N", "source": "secondary"}'), ('SYS-0093', '2025-12-21T06:02:38', '{"marker": "C", "source": "secondary"}'), ('SYS-0011', '2025-12-24T18:10:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0011', '2025-12-23T15:55:10', '{"marker": "L", "source": "secondary"}'), ('SYS-0143', '2025-12-19T06:03:27', '{"marker": "F", "source": "secondary"}'), ('SYS-0152', '2025-12-19T06:17:06', '{"marker": "K", "source": "secondary"}'), ('SYS-0015', '2025-12-23T15:55:14', '{"marker": "Q", "source": "secondary"}'), ('SYS-0001', '2025-12-23T06:02:00', '{"marker": "X", "source": "primary"}'), ('SYS-0209', '2025-12-19T06:23:09', '{"marker": "H", "source": "secondary"}'), ('SYS-0133', '2025-12-19T06:03:17', '{"marker": "C", "source": "secondary"}'), ('SYS-0158', '2025-12-19T06:17:12', '{"marker": "V", "source": "secondary"}'), ('SYS-0195', '2025-12-19T06:17:46', '{"marker": "D", "source": "secondary"}'), ('SYS-0182', '2025-12-19T06:17:33', '{"marker": "O", "source": "secondary"}'), ('SYS-0033', '2025-12-23T15:55:32', '{"marker": "W", "source": "secondary"}'), ('SYS-0083', '2025-12-21T06:02:28', '{"marker": "F", "source": "secondary"}'), ('SYS-0002', '2025-12-23T06:02:01', '{"marker": "Y", "source": "primary"}'), ('SYS-0076', '2025-12-23T06:02:22', '{"marker": "A", "source": "secondary"}'), ('SYS-0009', '2025-12-24T08:02:08', '{"marker": "U", "source": "primary"}'), ('SYS-0194', '2025-12-19T06:17:45', '{"marker": "U", "source": "secondary"}'); ``` ``` $ sqlite3 SQLite version 3.50.4 2025-07-30 19:33:53 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day15-inserts.sql Parse error near line 19: unrecognized token: ":" 6:02:26', '{"marker": "T", "source": "secondary"}'::jsonb), ('SYS-0137', '2025 error here ---^ Parse error near line 275: unrecognized token: ":" 5:55:12', '{"marker": "F", "source": "secondary"}'::jsonb), ('SYS-0002', '2025 error here ---^ sqlite> .read day15-inserts-sqlite.sql sqlite> .schema CREATE TABLE system_dispatches ( id SERIAL PRIMARY KEY, system_id TEXT NOT NULL, dispatched_at TIMESTAMP NOT NULL, payload JSONB NOT NULL, marker_letter TEXT GENERATED ALWAYS AS (json_extract(payload, '$.marker')) STORED, UNIQUE (system_id, dispatched_at, payload) ); CREATE TABLE incoming_dispatches ( system_id TEXT, dispatched_at TIMESTAMP, payload JSONB ); sqlite> .mode table sqlite> SELECT * FROM system_dispatches LIMIT 10; +----+-----------+---------------------+----------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+----------------------------------------+---------------+ | | SYS-0081 | 2025-12-21T06:02:26 | {"marker": "T", "source": "secondary"} | T | | | SYS-0137 | 2025-12-19T06:03:21 | {"marker": "D", "source": "secondary"} | D | | | SYS-0237 | 2025-12-19T06:23:37 | {"marker": "D", "source": "secondary"} | D | | | SYS-0006 | 2025-12-24T18:10:16 | {"marker": "K", "source": "secondary"} | K | | | SYS-0170 | 2025-12-19T06:17:24 | {"marker": "I", "source": "secondary"} | I | | | SYS-0224 | 2025-12-19T06:23:24 | {"marker": "S", "source": "secondary"} | S | | | SYS-0007 | 2025-12-24T18:10:06 | {"marker": "O", "source": "secondary"} | O | | | SYS-0035 | 2025-12-23T15:55:34 | {"marker": "C", "source": "secondary"} | C | | | SYS-0010 | 2025-12-23T15:55:09 | {"marker": "L", "source": "secondary"} | L | | | SYS-0037 | 2025-12-23T15:55:36 | {"marker": "D", "source": "secondary"} | D | +----+-----------+---------------------+----------------------------------------+---------------+ sqlite> SELECT * FROM incoming_dispatches LIMIT 10; +-----------+---------------------+----------------------------------------+ | system_id | dispatched_at | payload | +-----------+---------------------+----------------------------------------+ | SYS-0013 | 2025-12-23T15:55:12 | {"marker": "F", "source": "secondary"} | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | | SYS-0012 | 2025-12-23T15:55:11 | {"marker": "O", "source": "secondary"} | | SYS-0108 | 2025-12-19T06:02:52 | {"marker": "G", "source": "secondary"} | | SYS-0165 | 2025-12-19T06:17:19 | {"marker": "Q", "source": "secondary"} | | SYS-0189 | 2025-12-19T06:17:40 | {"marker": "H", "source": "secondary"} | | SYS-0050 | 2025-12-23T15:55:32 | {"marker": "K", "source": "secondary"} | | SYS-0138 | 2025-12-19T06:03:22 | {"marker": "V", "source": "secondary"} | | SYS-0087 | 2025-12-21T06:02:32 | {"marker": "A", "source": "secondary"} | | SYS-0208 | 2025-12-19T06:23:08 | {"marker": "G", "source": "secondary"} | +-----------+---------------------+----------------------------------------+ sqlite> ``` Good to go! ## Problem > Reconstruct the final confirmation phrase to text Santa based on the elves’ hazy recollection of how they solved this problem before. > > Your final result should include the marker_letter for each system, using only the most recent dispatch from a primary source. Once the correct dispatch has been identified for every system, combine the results and order them by dispatched_at in ascending order to reveal the confirmation phrase. > > The sleigh won’t launch without it. So, we need to first make sure we have the right `dispatches`. We have two tables. 1. `system_dispatches` 2. `incoming_dispatches` The problem statement also states: > The system kept throwing errors until we figured out how to handle duplicates. Whatever you do the records already in system_dispatches must take precedence.” So, we need to take the `system_dispatches` as the ground truth. However we need to include something or the other from the `incoming_dispatches` as there are new entries being created. So, we will first try to insert into the system dispatches everything in the `incoming_dispatches` ```sql INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; ``` ``` sqlite> INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; Runtime error: UNIQUE constraint failed: system_dispatches.system_id, system_dispatches.dispatched_at, system_dispatches.payload (19) sqlite> ``` Ops! ```sql SELECT COUNT(*) as system_dispatches_count FROM system_dispatches ; SELECT COUNT(*) as incoming_dispatches_count FROM incoming_dispatches; SELECT COUNT(*) as common_dispatches_count FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches INTERSECT SELECT * FROM incoming_dispatches); ``` ``` sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 254 | +----------+ sqlite> SELECT COUNT(*) FROM incoming_dispatches; +----------+ | COUNT(*) | +----------+ | 38 | +----------+ sqlite> SELECT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches UNION SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> SELECT DISTINCT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches UNION SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> SELECT COUNT(*) FROM (SELECT system_id, dispatched_at, payload FROM system_dispatches INTERSECT SELECT * FROM incoming_dispatches); +----------+ | COUNT(*) | +----------+ | 10 | +----------+ sqlite> ``` We need to be careful about what we insert into the `system_dispatches`. I think the duplicates causes this conflict in constraints, we can ignore on the constraint failures and don't insert that row. ```sql INSERT OR IGNORE INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; ``` Ok, now if we see the count, we get `282` which is making sense. - The original `system_dispatches` had a count of `254` before merge. - The original `incoming_dispatches` had a count of `38`. - After merging `incoming_dispatches` into `system_dispatches` table, we have a `282` records in `system_dispatches` but `254+38=292`, what about the `10`? Those were already there in the `system_dispatches`. As we saw in the earlier query, there were `10` rows in common from the `incoming_dispatches` table before merge. ``` sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 254 | +----------+ sqlite> SELECT COUNT(*) FROM incoming_dispatches; +----------+ | COUNT(*) | +----------+ | 38 | +----------+ sqlite> INSERT INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; Runtime error: UNIQUE constraint failed: system_dispatches.system_id, system_dispatches.dispatched_at, system_dispatches.payload (19) sqlite> INSERT OR IGNORE INTO system_dispatches (system_id, dispatched_at, payload) SELECT system_id, dispatched_at, payload FROM incoming_dispatches; sqlite> SELECT COUNT(*) FROM system_dispatches ; +----------+ | COUNT(*) | +----------+ | 282 | +----------+ sqlite> ``` Now, to the actual problem. What we need to do now? - Using only the most recent dispatch from a primary source. - Once the correct dispatch has been identified for every system, combine the results and order them by dispatched_at in ascending order - Reveal the confirmation phrase We have the single source now! ```sql SELECT * FROM system_dispatches LIMIT 10; ``` ``` SELECT * FROM system_dispatches LIMIT 10; +----+-----------+---------------------+----------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+----------------------------------------+---------------+ | | SYS-0081 | 2025-12-21T06:02:26 | {"marker": "T", "source": "secondary"} | T | | | SYS-0137 | 2025-12-19T06:03:21 | {"marker": "D", "source": "secondary"} | D | | | SYS-0237 | 2025-12-19T06:23:37 | {"marker": "D", "source": "secondary"} | D | | | SYS-0006 | 2025-12-24T18:10:16 | {"marker": "K", "source": "secondary"} | K | | | SYS-0170 | 2025-12-19T06:17:24 | {"marker": "I", "source": "secondary"} | I | | | SYS-0224 | 2025-12-19T06:23:24 | {"marker": "S", "source": "secondary"} | S | | | SYS-0007 | 2025-12-24T18:10:06 | {"marker": "O", "source": "secondary"} | O | | | SYS-0035 | 2025-12-23T15:55:34 | {"marker": "C", "source": "secondary"} | C | | | SYS-0010 | 2025-12-23T15:55:09 | {"marker": "L", "source": "secondary"} | L | | | SYS-0037 | 2025-12-23T15:55:36 | {"marker": "D", "source": "secondary"} | D | +----+-----------+---------------------+----------------------------------------+---------------+ sqlite> ``` We have 4 columns. - system_id - dispatched_at - payload (json string) - marker (its a generated table from the payload["marker"] string) So, we need to group by system, that's where the `system_id` is for. Then we need to order by `dispatched_at` with the latest ones. Finally we also need to filter the records that has `payload["source"]` as `primary`. But wait, if we group by `system_id` then how can we get the latest dispatched ordered correctly? We can't use it in the `ORDER BY` as the group would have already been created right? Oh! We might need window functions, but just a moment, can we do it without them? Let's try. ```sql SELECT system_id, MAX(dispatched_at), payload, marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` Oh boy, that was simple! We just grouped by `system_id`, filtered the source as `primary` and selected the row with `MAX(dispatched_at)` which will give us the latest dispatch record for a system. Boom! The confirmation phrase is `ADVENTOFSQL` We can just select the `marker_letter` ```sql SELECT marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` ``` sqlite> SELECT marker_letter FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; +---------------+ | marker_letter | +---------------+ | X | | Y | | Z | | M | | P | | R | | K | | B | | U | | C | | J | +---------------+ sqlite> ``` Whops! What happened, the marker letter changed? Yep, because we have ordered by `dispatched_at DESC` but never told how to return the rows, by using `MAX(dispatched_at)` we were selecting the latest date record. Without it, we are selecting the first record in the group that could be the first or the oldest dispatched_at time. So, we need `MAX(dispatched_at)` included in the selected result set. ```sql SELECT marker_letter, MAX(dispatched_at) FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; ``` ``` SELECT marker_letter, MAX(dispatched_at) FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' GROUP BY system_id ORDER BY dispatched_at ASC; +---------------+---------------------+ | marker_letter | MAX(dispatched_at) | +---------------+---------------------+ | A | 2025-12-24T08:10:00 | | D | 2025-12-24T08:10:01 | | V | 2025-12-24T08:10:02 | | E | 2025-12-24T08:10:03 | | N | 2025-12-24T08:10:04 | | T | 2025-12-24T08:10:05 | | O | 2025-12-24T08:10:06 | | F | 2025-12-24T08:10:07 | | S | 2025-12-24T08:10:08 | | Q | 2025-12-24T08:10:09 | | L | 2025-12-24T08:10:10 | +---------------+---------------------+ sqlite> ``` There, we go! It scared me for a moment. We order by, only keep the `source` as `primary` and then select the latest `dispatched_at` for a given system_id. If you don't want to do this way! I found a few more hacks ### With Subquery We can SELECT the `MAX(dispatched_at)` in a subquery for that system_id and filter based on the `primary` source as usual. ```sql SELECT * FROM system_dispatches WHERE json_extract(system_dispatches.payload, '$.source') = 'primary' AND system_dispatches.dispatched_at = ( SELECT MAX(latest_dispatches.dispatched_at) FROM system_dispatches latest_dispatches WHERE latest_dispatches.system_id = system_dispatches.system_id AND json_extract(latest_dispatches.payload, '$.source') = 'primary' ) ORDER BY system_dispatches.dispatched_at ASC; ``` ``` sqlite> SELECT * FROM system_dispatches WHERE json_extract(system_dispatches.payload, '$.source') = 'primary' AND system_dispatches.dispatched_at = ( SELECT MAX(latest_dispatches.dispatched_at) FROM system_dispatches latest_dispatches WHERE latest_dispatches.system_id = system_dispatches.system_id AND json_extract(latest_dispatches.payload, '$.source') = 'primary' ) ORDER BY system_dispatches.dispatched_at ASC; +----+-----------+---------------------+--------------------------------------+---------------+ | id | system_id | dispatched_at | payload | marker_letter | +----+-----------+---------------------+--------------------------------------+---------------+ | | SYS-0001 | 2025-12-24T08:10:00 | {"marker": "A", "source": "primary"} | A | | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | D | | | SYS-0003 | 2025-12-24T08:10:02 | {"marker": "V", "source": "primary"} | V | | | SYS-0004 | 2025-12-24T08:10:03 | {"marker": "E", "source": "primary"} | E | | | SYS-0005 | 2025-12-24T08:10:04 | {"marker": "N", "source": "primary"} | N | | | SYS-0006 | 2025-12-24T08:10:05 | {"marker": "T", "source": "primary"} | T | | | SYS-0007 | 2025-12-24T08:10:06 | {"marker": "O", "source": "primary"} | O | | | SYS-0008 | 2025-12-24T08:10:07 | {"marker": "F", "source": "primary"} | F | | | SYS-0009 | 2025-12-24T08:10:08 | {"marker": "S", "source": "primary"} | S | | | SYS-0010 | 2025-12-24T08:10:09 | {"marker": "Q", "source": "primary"} | Q | | | SYS-0011 | 2025-12-24T08:10:10 | {"marker": "L", "source": "primary"} | L | +----+-----------+---------------------+--------------------------------------+---------------+ sqlite> ``` ### Window Funciton We can even use the window function to partition by the `system_id` and there itself order by the `dispatched_at` latest time. And then select it as a CTE table. We can use a `ROW_NUMBER` function to assign a rank to each row per system_id and ordered by the latest `dispatched_at` ```sql WITH latest_dispatches AS ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY system_id ORDER BY dispatched_at DESC ) as rank FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' ) SELECT * FROM latest_dispatches WHERE rank = 1 ORDER BY dispatched_at ASC; ``` The rest is the same as the above query. We order by dispatched latest but then order by ascending (oldest first) in the final result set. ``` sqlite> WITH latest_dispatches AS ( SELECT *, ROW_NUMBER() OVER ( PARTITION BY system_id ORDER BY dispatched_at DESC ) as rank FROM system_dispatches WHERE json_extract(payload, '$.source') = 'primary' ) SELECT * FROM latest_dispatches WHERE rank = 1 ORDER BY dispatched_at ASC; +----+-----------+---------------------+--------------------------------------+---------------+------+ | id | system_id | dispatched_at | payload | marker_letter | rank | +----+-----------+---------------------+--------------------------------------+---------------+------+ | | SYS-0001 | 2025-12-24T08:10:00 | {"marker": "A", "source": "primary"} | A | 1 | | | SYS-0002 | 2025-12-24T08:10:01 | {"marker": "D", "source": "primary"} | D | 1 | | | SYS-0003 | 2025-12-24T08:10:02 | {"marker": "V", "source": "primary"} | V | 1 | | | SYS-0004 | 2025-12-24T08:10:03 | {"marker": "E", "source": "primary"} | E | 1 | | | SYS-0005 | 2025-12-24T08:10:04 | {"marker": "N", "source": "primary"} | N | 1 | | | SYS-0006 | 2025-12-24T08:10:05 | {"marker": "T", "source": "primary"} | T | 1 | | | SYS-0007 | 2025-12-24T08:10:06 | {"marker": "O", "source": "primary"} | O | 1 | | | SYS-0008 | 2025-12-24T08:10:07 | {"marker": "F", "source": "primary"} | F | 1 | | | SYS-0009 | 2025-12-24T08:10:08 | {"marker": "S", "source": "primary"} | S | 1 | | | SYS-0010 | 2025-12-24T08:10:09 | {"marker": "Q", "source": "primary"} | Q | 1 | | | SYS-0011 | 2025-12-24T08:10:10 | {"marker": "L", "source": "primary"} | L | 1 | +----+-----------+---------------------+--------------------------------------+---------------+------+ sqlite> ``` Phew! That was a good one to end the advent of SQL! I enjoyed it! I learnt a ton - CTEs - JOINs (some wired stuff can be done) - Window Functions (LAG, LEAD, ROW_NUMBER) - FTS (in SQLite) - Json parsing - date shenanigans - CTEs don't support insert and delete in SQLite (it ruined my day 10 solution) - Recursive CTEs - String manipulation (thanks to xml parsing) - I need to write a post about explaining what I learned. Thanks [Aaron francis](https://aaronfrancis.com/) from [databaseschool](https://databaseschool.com/) for this challenge and explaining in depth on each day(i didn't watch all) and [Kelsey Petrich](https://x.com/krpetrich) for the lore of each problem, those were really lovely to read! Happy Coding :) Merry Christmas Happy New year Whatever you celebrate! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-14 Advent of SQL, Day 14 - Ski Resort Paths Ok, almost to the penultimate day in the series. It is day 14 of Advent of SQL. Let's grab the SQL for the day. That's Sun, 28 Dec 2025 00:00:00 UTC ## Advent of SQL, Day 14 - Ski Resort Paths Ok, almost to the penultimate day in the series. It is day 14 of Advent of SQL. Let's grab the SQL for the day. ```sql DROP TABLE IF EXISTS mountain_network; CREATE TABLE mountain_network ( id INTEGER PRIMARY KEY, from_node TEXT, to_node TEXT, node_type TEXT, -- 'Lift' or 'Trail' difficulty TEXT -- Only applicable for trails: 'green', 'blue', 'black', 'double_black' ); INSERT INTO mountain_network (id, from_node, to_node, node_type, difficulty) VALUES (1, 'Outlaw Express', 'Stairway Lift', 'Lift', NULL), (2, 'Outlaw Express', 'Top Gun Bowl', 'Trail', 'black'), (3, 'Top Gun Bowl', 'Top Gun', 'Trail', 'black'), (4, 'Top Gun', 'Montoya', 'Trail', 'blue'), (5, 'Montoya', 'Center Aisle', 'Trail', 'green'), (6, 'Center Aisle', 'Lower Stampede', 'Trail', 'green'), (7, 'Stairway Lift', 'Red''s Lift', 'Lift', NULL), (8, 'Stairway Lift', 'Broadway', 'Trail', 'green'), (9, 'Red''s Lift', 'Bearclaw', 'Trail', 'blue'), (10, 'Bearclaw', 'Last Chance', 'Trail', 'blue'), (11, 'Last Chance', 'Diamondback', 'Trail', 'blue'), (12, 'Diamondback', 'Broadway', 'Trail', 'green'), (13, 'Red''s Lift', 'Bishop''s Bowl', 'Trail', 'black'), (14, 'Red''s Lift', 'Amy''s Ridge', 'Trail', 'blue'), (15, 'Amy''s Ridge', 'Grizzly Bowl', 'Trail', 'black'), (16, 'Flathead Lift', 'Amy''s Ridge', 'Trail', 'blue'), (17, 'Jake''s Lift', 'Wildwood Lift', 'Lift', NULL), (18, 'Wildwood Lift', 'Sidewinder', 'Trail', 'green'), (19, 'Wildwood Lift', 'Brightside', 'Trail', 'blue'), (20, 'Brightside', 'Moonrise', 'Trail', 'green'), (21, 'Moonrise', 'Draw', 'Trail', 'green'), (22, 'Moonrise', 'Lone Pine', 'Trail', 'blue'), (23, 'Draw', 'Maverick', 'Trail', 'blue'), (24, 'Draw', 'Broadway', 'Trail', 'green'), (25, 'Broadway', 'Outlaw Trail', 'Trail', 'green'), (26, 'Outlaw Trail', 'Center Aisle', 'Trail', 'green'), (27, 'Center Aisle', 'Bandit', 'Trail', 'green'), (28, 'Jake''s Lift', 'Maverick', 'Trail', 'blue'); ``` That's it, we just have 1 table called `mountain_network` and it has `28` records. Let's look at the problem statement to check what we need to do to make sense of those 28 rows. ## Problem > Find all the possible routes from Jake's Lift to Maverick. None of the possible routes will take more than 12 connections. Oh! This is a graph like or network like problem. Ouch! Relational databases looks like they would be a good fit for these stuff, but if the data is just in the single table, there's not really much help. Let's see how we can think about it. ### JOINs and UNIONS We need to find all the routes from a start node(record) i.e. `Jake's Lift` and find all ways that lead to the node(record) i.e. `Maverick`. The table gives a list of edges i.e. from which node to which node there is a way or a trail or lift. So, we can do a simple select to check if the `from_node` has `Jake's Lift` and the `to_node` has `Maverick` right? But that would make it too long. Like we would have to look for 12 consecutive edges and branch off at each direction. For the first level it would look simple like this: ```sql SELECT mountain_network.from_node || ' -> ' || mountain_network.to_node AS path, 1 AS connections FROM mountain_network WHERE mountain_network.from_node = 'Jake''s Lift' AND mountain_network.to_node = 'Maverick'; ``` ``` sqlite> SELECT mountain_network.from_node || ' -> ' || mountain_network.to_node AS path, 1 AS connections FROM mountain_network WHERE mountain_network.from_node = 'Jake''s Lift' AND mountain_network.to_node = 'Maverick' ...> ; +-------------------------+-------------+ | path | connections | +-------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +-------------------------+-------------+ sqlite> ``` Luckily, we have one route directly from `Jake's Lift` to `Maverick`. But that might not be for all levels. Like we need to check then, from each of the start nodes that begin at `Jake's Lift`, then for all the nodes that begin at that then it branches of further till we have a dead end only at `Maverick`. Phew that's going to be a long one. ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node, 2 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift' AND T2.to_node = 'Maverick'; ``` That is the first branch off from `Jake's Lift`. If we remove the `T2.to_node = 'Maverick'` you would see all nodes from the `Jake's Lift`, not necessarily ending at `Maverick` ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node as path, 2 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift'; ``` ``` sqlite> SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node as path, 2 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift'; +--------------------------------------------+-------------+ | path | connections | +--------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside | 2 | | Jake's Lift -> Wildwood Lift -> Sidewinder | 2 | +--------------------------------------------+-------------+ sqlite> ``` Similarly we can do it for 3 and 4, and till 12 connections. ```sql SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node as path, 3 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift'; ``` ``` sqlite> SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node as path, 3 as connections FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift' ...> ; +--------------------------------------------------------+-------------+ | path | connections | +--------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise | 3 | +--------------------------------------------------------+-------------+ sqlite> ``` Yikes. That gives a long long query. I cannot write. ```sql SELECT T1.from_node || ' -> ' || T1.to_node AS path, 1 AS connections FROM mountain_network T1 WHERE T1.from_node = 'Jake''s Lift' AND T1.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node, 2 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node WHERE T1.from_node = 'Jake''s Lift' AND T2.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node, 3 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node WHERE T1.from_node = 'Jake''s Lift' AND T3.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node, 4 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node WHERE T1.from_node = 'Jake''s Lift' AND T4.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node, 5 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node WHERE T1.from_node = 'Jake''s Lift' AND T5.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node, 6 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node WHERE T1.from_node = 'Jake''s Lift' AND T6.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node, 7 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node WHERE T1.from_node = 'Jake''s Lift' AND T7.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node, 8 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node WHERE T1.from_node = 'Jake''s Lift' AND T8.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node, 9 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node WHERE T1.from_node = 'Jake''s Lift' AND T9.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node, 10 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node WHERE T1.from_node = 'Jake''s Lift' AND T10.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node || ' -> ' || T11.to_node, 11 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node JOIN mountain_network T11 ON T10.to_node = T11.from_node WHERE T1.from_node = 'Jake''s Lift' AND T11.to_node = 'Maverick' UNION ALL SELECT T1.from_node || ' -> ' || T1.to_node || ' -> ' || T2.to_node || ' -> ' || T3.to_node || ' -> ' || T4.to_node || ' -> ' || T5.to_node || ' -> ' || T6.to_node || ' -> ' || T7.to_node || ' -> ' || T8.to_node || ' -> ' || T9.to_node || ' -> ' || T10.to_node || ' -> ' || T11.to_node || ' -> ' || T12.to_node, 12 FROM mountain_network T1 JOIN mountain_network T2 ON T1.to_node = T2.from_node JOIN mountain_network T3 ON T2.to_node = T3.from_node JOIN mountain_network T4 ON T3.to_node = T4.from_node JOIN mountain_network T5 ON T4.to_node = T5.from_node JOIN mountain_network T6 ON T5.to_node = T6.from_node JOIN mountain_network T7 ON T6.to_node = T7.from_node JOIN mountain_network T8 ON T7.to_node = T8.from_node JOIN mountain_network T9 ON T8.to_node = T9.from_node JOIN mountain_network T10 ON T9.to_node = T10.from_node JOIN mountain_network T11 ON T10.to_node = T11.from_node JOIN mountain_network T12 ON T11.to_node = T12.from_node WHERE T1.from_node = 'Jake''s Lift' AND T12.to_node = 'Maverick'; ``` ``` +--------------------------------------------------------------+-------------+ | path | connections | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | aw -> Maverick | | +--------------------------------------------------------------+-------------+ sqlite> ``` Ok, we get the result, but that looks like a terrifying query. Can we do better? I looked at the tutorial and we have something like `RECURSIVE CTE` Wow! This is quite challenging to explain. ### Recursive CTE We can create a recursive CTE that draws the path until we have not found the stop node that is, `Maverick` and the number of connections is not more than `12`. So here's how it goes. ```sql WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT full_path, connections FROM ski_paths WHERE to_node = 'Maverick' ORDER BY connections ASC; ``` We first define the base case i.e. what to select first. We get the first row where we start, the `Jake's Lift` and from there on we create a recursive select statement by referencing the CTE within it. The recursive bit is this one ```sql SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ``` We are referencing the CTE (table) within it as the recursive condition to select from the nodes where the end node or the `to_node` is not Maverick and connections haven't gone beyond `12`, we try to find the path. This would give us all the nodes. Let's see first, by checking the CTE as is. ```sql WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT * from ski_paths; ``` ``` sqlite> WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT * from ski_paths; +---------------+----------------+--------------------------------------------------------------+-------------+ | from_node | to_node | full_path | connections | +---------------+----------------+--------------------------------------------------------------+-------------+ | Jake's Lift | Wildwood Lift | Jake's Lift -> Wildwood Lift | 1 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Jake's Lift | Maverick | Jake's Lift -> Maverick | 1 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Wildwood Lift | Brightside | Jake's Lift -> Wildwood Lift -> Brightside | 2 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Wildwood Lift | Sidewinder | Jake's Lift -> Wildwood Lift -> Sidewinder | 2 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Brightside | Moonrise | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise | 3 | +---------------+----------------+--------------------------------------------------------------+-------------+ | Moonrise | Draw | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 4 | | | | aw | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Moonrise | Lone Pine | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Lo | 4 | | | | ne Pine | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Draw | Broadway | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | | | aw -> Broadway | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Draw | Maverick | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | | | aw -> Maverick | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Broadway | Outlaw Trail | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 6 | | | | aw -> Broadway -> Outlaw Trail | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Outlaw Trail | Center Aisle | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 7 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Center Aisle | Bandit | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 8 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle -> Bandit | | +---------------+----------------+--------------------------------------------------------------+-------------+ | Center Aisle | Lower Stampede | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 8 | | | | aw -> Broadway -> Outlaw Trail -> Center Aisle -> Lower Stam | | | | | pede | | +---------------+----------------+--------------------------------------------------------------+-------------+ sqlite> ``` You can see we have got quite a lot of paths, this are all the paths that start from `Jake's Lift` and have less than `12` connections. So, now we can simply filter with the `WHERE` clause in the case where the `to_node = 'Maverick'` and we would get the result. ``` sqlite> WITH RECURSIVE ski_paths AS ( SELECT from_node, to_node, CAST(from_node || ' -> ' || to_node AS TEXT) AS full_path, 1 AS connections FROM mountain_network WHERE from_node = 'Jake''s Lift' UNION ALL SELECT mn.from_node, mn.to_node, sp.full_path || ' -> ' || mn.to_node, sp.connections + 1 FROM ski_paths sp JOIN mountain_network mn ON sp.to_node = mn.from_node WHERE sp.connections < 12 AND sp.to_node != 'Maverick' ) SELECT full_path, connections FROM ski_paths WHERE to_node = 'Maverick' ORDER BY connections ASC; +--------------------------------------------------------------+-------------+ | full_path | connections | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Maverick | 1 | +--------------------------------------------------------------+-------------+ | Jake's Lift -> Wildwood Lift -> Brightside -> Moonrise -> Dr | 5 | | aw -> Maverick | | +--------------------------------------------------------------+-------------+ sqlite> ``` Ok, that gave the result correctly. And this is it! Sweet problem to learn about Recursive CTE! Off to the final day of advent of SQL 2025! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-12 Advent of SQL - Day 12, Archive Flight Records We are on Day 12! Phew its almost done! Just 3 days more! Let's get the SQL! We have just one table and a couple Sat, 27 Dec 2025 00:00:00 UTC ## Advent of SQL - Day 12, Archive Flight Records We are on Day 12! Phew its almost done! Just 3 days more! Let's get the SQL! ```sql DROP TABLE IF EXISTS archive_records; CREATE TABLE archive_records ( id INT PRIMARY KEY, title TEXT, description TEXT ); INSERT INTO archive_records (id, title, description) VALUES (1, 'Flight Stabilization Prototype Analysis', 'This report details the latest advancements in stabilizing aerial maneuvers for enchanted sleighs. Initial tests yielded promising results, showcasing a marked decrease in turbulence during airborne navigation.'), (2, 'Lift Calibration Incident Log', 'During a routine lift calibration test, the sleigh experienced an unexpected upward surge, causing it to hover dangerously close to the workshop ceiling. Subsequent analysis revealed a miscalculation in the weight distribution formula, prompting a thorough review of all aerodynamic coefficients.'), (3, 'Aerial Aspirations: The Great Flop', 'Despite the initial excitement surrounding the design of our feather-laden airborne contraption, the prototype proved less than buoyant. The unexpected descent resulted in a rather spectacular cloud of glitter and twigs, serving as a vivid reminder that not all dreams of flight take wing as intended.'), (4, 'Optimized Sled Dynamics', 'This document explores advanced mechanics behind sled propulsion and movement efficiency. Through extensive calculations and enchanted material tests, the elves aim to refine turning capabilities and enhance downhill speed.'), (5, 'Reindeer Harness Design Flaws', 'The latest prototype of our reindeer harness was found to inhibit mobility, causing undue strain on the animals during testing. Observations indicated that the weight distribution was poorly calibrated, necessitating a complete redesign for optimal comfort and performance.'), (6, 'Streamlined Cargo Routing System', 'In our latest endeavor, we have implemented a magical algorithm to optimize the route taken by our toy-laden sleighs. This groundbreaking update minimizes transport time between the workshop and delivery points, ensuring that holiday cheer reaches every home even faster!'), (7, 'Toy Durability Testing Protocols', 'The elves meticulously conducted stress evaluations on the latest toy prototypes to determine their resilience under various conditions. Initial findings indicate that while some designs withstood rigorous play, others required reinforcements to avoid premature wear and tear.'), (8, 'Weather Resistance Breakthrough: Frost Shielding', 'In our recent experiments, we discovered an innovative composite material that effectively repels moisture while withstanding extreme cold. This newfound frost shielding could revolutionize our outdoor toys, ensuring they remain both functional and enchanting, even in the harshest winter conditions.'), (9, 'Safety Compliance Check Overview', 'In the pursuit of enchantment and joy, this document outlines the mandatory safety compliance measures for all workshop operations. Each elf must adhere strictly to these guidelines to ensure the safe transport and handling of our delicate prototypes, thus preventing any unforeseen magical mishaps.'), (10, 'Sleigh Skim Mechanism Upgrade', 'The experiment involved enchanting the underside of the sleigh with a whispering wind charm to achieve unprecedented speeds. Results were alarming, leading to uncontrollable flight trajectories and sudden descents—definitely do not attempt again.'), (11, 'Caution: Enchanted Toy Prototype', 'This design incorporates a reactive magic component that may unpredictably animate in the presence of mischief. Ensure all test environments are secured against spontaneous giggles and potential chaos.'), (12, 'Workshop Experiment Safety Checklist', 'Before embarking on any workshop experiments, ensure all safety goggles are securely fastened to prevent debris from interfered visions. Always double-check that the workspace is free of clutter, as unexpected accidents can arise from even the smallest flurry of trinkets and tools.'), (13, 'Cocoa Bean Roasting Innovations', 'This review explores the latest techniques in roasting cocoa beans to achieve unparalleled flavor profiles. Adjustments to temperature and timing have led to a delightful spectrum of aromas, promising to elevate our confectionery creations to new heights.'), (14, 'Intricate Snowflake Ornament Design', 'This design blueprint outlines the geometric intricacies of a multi-faceted snowflake ornament, emphasizing a balance between elegance and structural integrity. Each arm is meticulously patterned to reflect light, creating a shimmering effect that dances with the seasons, while ensuring optimal symmetry for enchanting visual appeal.'), (15, 'Gift Box Assembly Prototype', 'This prototype outlines the intricate process of assembling the enchanted gift boxes designed to withstand the whims of time and space. Each step must be meticulously executed to ensure that every box not only sparkles with joy but also maintains its magical properties through every unwrapping.'), (16, 'Wrap-It-Up: Innovative Designs', 'This experimental report explores various materials and techniques for creating enchanted wrapping paper that enhances the gift-giving experience. Initial findings suggest that incorporation of shimmering elven dust can amplify the aesthetic appeal while maintaining structural integrity during airborne delivery.'), (17, 'Magical Confection Fusion Results', 'The experimental concoction blended sugar crystals with essence of starlight, resulting in a luminescent treat that sparkles enchantingly. However, a curious side effect was noted: excessive giggling among taste testers, raising questions about potential airborne laughter.'), (18, 'Luminous Ornament Crafting Techniques', 'In our continuous pursuit of radiance, this document outlines innovative methods for creating ornaments that glow with enchantment. Engaging both traditional techniques and modern enchantments, each design is meant to instill joy and sparkle during the festive season.'), (19, 'Magical Energy Conduction Analysis', 'Recent experiments have shown that the flow of magical energy through crystalline conduits behaves unpredictably under varying lunar phases. Further investigation into the correlation between ambient mana levels and energy stability is necessary to optimize enchantment potency.'), (20, 'Elven Workshop Organization Protocols', 'The implementation of open shelving systems has significantly increased accessibility to essential materials, thus enhancing workflow efficiency. Furthermore, the organization of tools into color-coded bins ensures that each elf can swiftly locate their required implements without disrupting the harmony of the workshop.'); ``` We have just one table and a couple of text like columns. That's it, looks like a string searching problem. Let's head to the problem statement! ## Problem > Using the `archive_records` table, search both the `title` and `description` fields for the term "fly". Make sure that you also match for words like "flying", "flight", etc. Boost the results where the term appears in the title and lastly, rank the results by relevance (most relevant first). Provide the elves the top 5 most relevant archived records back. It is a text search use case problem indeed! We need to find and rank the records matching the word `fly`, `flying`, `flight` like those. Kind of tricky if we miss any terms that are not hard-coded. We need to boost the search term in `title`, so there is more weightage if the term appears in title than in description. Makes sense! Let's start simple and move into full-text-search in SQLite! ### Simple String Matching We start by a simple nested `CASE WHEN THEN` condition. We check if the `title` has `fly`, `flying`, `flight`, etc then we set the rank as `2` and add the score from `description` as `1` if the same terms appear in description. So, - If the search term (fly, flight, etc) appears **only** in title, score is `2` - If the search term appears **only** in description, score is `1` - If the search term appears in **both** title and description, the score is `3` since we are adding the scores. - If the search term doesn't appear at all, then the score remains `0`. We simply assign the score based on the appearance of the search term and then order the result based on the computed `rank` and list the top 5. ```sql SELECT id, title, description, ( CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 2 ELSE 0 END + CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_records ORDER BY rank DESC, id ASC LIMIT 5; ``` However, we can limit the search space by eliminating the computation of ranking all the records with a `WHERE` clause since it can optimise only the relevant records before giving out the result set. Hence, we filter only in the cases where the `title` and `description` have the relevant word and then use the condition to order by the `rank` score that we computed and list the top `5` ones. ```sql SELECT id, title, description, ( -- Title matches worth 2 points (boosted relevance) CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 2 ELSE 0 END + -- Description matches worth 1 point CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ORDER BY rank DESC, id ASC LIMIT 5; ``` We can wrap it in a CTE, but that doesn't does anything differently though. With CTE as below, we assign a common score like `1` or `0` based on the `title` or description match. Then in the outer query, we can assign a weight to the rank if the `title` matched as `2` or `description` matched as `1` and add it up to get the same `rank` for that record. ```sql WITH search_results AS ( SELECT id, title, description, CASE WHEN LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' THEN 1 ELSE 0 END AS title_match, CASE WHEN LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' THEN 1 ELSE 0 END AS desc_match FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ) SELECT id, title, description, (title_match * 2 + desc_match * 1) AS rank FROM search_results ORDER BY rank DESC, id ASC LIMIT 5; ``` ### Assign score based on matching frequency We can also make it better by counting how many times the term appeared in the relevant column and then compute a cumulative rank. This way it becomes more aggressive on the most relevant docs only. So, we write something like this. - Compute the length of the full column example as title with `LENGTH(LOWER(title)` then subtract the length of the word left after removing `fly`, or `flight`, or others. It calculates the difference in length between the original title and the one with 'fly' removed. Each 'fly' is 3 characters, so the difference divided by 3 gives the number of occurrences of 'fly'. This `(LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 ` counts how many times 'fly' appears. Similarly for `flight` and `flying` it is `6`, and so on. - If that appears in title it is multiplied by `2` - If it appears in description, it is kept as is (multiplied by `1`) but you can weight it accordingly, as there could be multiple column to weigh. ```sql SELECT id, title, description, ( ((LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flight', ''))) / 6 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flying', ''))) / 6 ) * 2 + ((LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'fly', ''))) / 3 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flight', ''))) / 6 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flying', ''))) / 6 ) ) AS rank FROM archive_records ORDER BY rank DESC, id ASC LIMIT 5; ``` You can always add a where clause to reduce the search space upfront: ```sql -- Simplified frequency-based ranking SELECT id, title, description, ( ((LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'fly', ''))) / 3 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flight', ''))) / 6 + (LENGTH(LOWER(title)) - LENGTH(REPLACE(LOWER(title), 'flying', ''))) / 6 ) * 2 + ((LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'fly', ''))) / 3 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flight', ''))) / 6 + (LENGTH(LOWER(description)) - LENGTH(REPLACE(LOWER(description), 'flying', ''))) / 6 ) ) AS rank FROM archive_records WHERE LOWER(title) LIKE '%fly%' OR LOWER(title) LIKE '%flight%' OR LOWER(title) LIKE '%flying%' OR LOWER(description) LIKE '%fly%' OR LOWER(description) LIKE '%flight%' OR LOWER(description) LIKE '%flying%' ORDER BY rank DESC, id ASC LIMIT 5; ``` This would give us similar results but slightly different as we are assigning rank on the number of times each term appears and not just if it appears once or not. We can even make the list of words and their weight as a CTE and dynamically use it in the actual query: ```sql WITH keywords AS ( SELECT 'fly' AS term, 2.0 AS title_weight, 1.0 AS desc_weight UNION ALL SELECT 'flight', 2.0, 1.0 UNION ALL SELECT 'flying', 2.0, 1.0 ), ranked AS ( SELECT a.id, a.title, a.description, SUM( ((LENGTH(LOWER(a.title)) - LENGTH(REPLACE(LOWER(a.title), k.term, ''))) / LENGTH(k.term)) * k.title_weight + ((LENGTH(LOWER(a.description)) - LENGTH(REPLACE(LOWER(a.description), k.term, ''))) / LENGTH(k.term)) * k.desc_weight ) AS rank FROM archive_records a CROSS JOIN keywords k GROUP BY a.id ) SELECT id, title, description, rank FROM ranked ORDER BY rank DESC, id ASC LIMIT 5; ``` Here we have 2 CTEs now: - `keywords` defining the words to search for and its weight - `ranked` defining the computed sum of weights with frequency on the relevant columns. As you can see, by just adding the keyword in the CTE along with its weight, the rest of the query can work without changing anything. ### Full Text Search We can now also look at the [FTS](https://sqlite.org/fts5.html) or Full Text Search in SQLite. In SQLite, we can create a `VIRTUAL TABLE` which is like - A table computed on the fly - Doesn't exist as physical entity in the database We have [fts5](https://sqlite.org/fts5.html) which we can use to match, equality, or expressions in functions to search for text in columns. ```sql CREATE VIRTUAL TABLE IF NOT EXISTS archive_fts USING fts5( title, description, content=archive_records ); ``` We need to provide the columns that we want to search against. In this case we want to search for `title` and `description` as the column. The `content=archive_records`, links the FTS table to a real table `archive_records` instead of storing its own copy of all text, the FTS table indexes the data from archive_records. Then we want to insert all the records from the `archive_records` for it to make aware of the records existing in the actual table. ```sql INSERT OR IGNORE INTO archive_fts(rowid, title, description) SELECT id, title, description FROM archive_records; ``` Then we can query it like so: ```sql SELECT * FROM archive_fts WHERE archive_fts MATCH 'fly'; ``` We have a couple of option to search against like 1. MATCH 2. Boolean Operator 3. Expressions and Wildcards We simply can filter the records with this ```sql SELECT * FROM archive_fts JOIN archive_records ON archive_fts.rowid = archive_records.id WHERE archive_fts MATCH 'fly OR flight OR flying'; ``` We have added `MATCH 'fly OR flight OR flying'` to only limit the search space on those keywords. ```sql SELECT archive_records.id, archive_records.title, archive_records.description, ( CASE WHEN archive_records.title LIKE '%fly%' OR archive_records.title LIKE '%flight%' OR archive_records.title LIKE '%flying%' THEN 2 ELSE 0 END + CASE WHEN archive_records.description LIKE '%fly%' OR archive_records.description LIKE '%flight%' OR archive_records.description LIKE '%flying%' THEN 1 ELSE 0 END ) AS rank FROM archive_fts JOIN archive_records ON archive_fts.rowid = archive_records.id WHERE archive_fts MATCH 'fly OR flight OR flying' ORDER BY rank DESC, archive_records.id ASC LIMIT 5; ``` This simply will give us the relevant results by doing a fuzzy search which is way better then `LIKE` or `%` wildcard operators in the columns. So, that are the approaches I like taking on day 12. It was fun! Working with full text search for the first time. On to day 13! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-13 Advent of SQL - Day 13, XML Travel Manifests Its day 13 of Advent of SQL, we have some xml to parse, which I don't think SQL can handle, but string manipulation Sat, 27 Dec 2025 00:00:00 UTC ## Advent of SQL - Day 13, XML Travel Manifests Its day 13 of Advent of SQL, we have some xml to parse, which I don't think SQL can handle, but string manipulation to the rescue. Let's get the SQL for the day: ```sql DROP TABLE IF EXISTS travel_manifests; CREATE TABLE travel_manifests ( manifest_id INT PRIMARY KEY, vehicle_id TEXT, departure_time TIMESTAMP, manifest_xml XML ); INSERT INTO travel_manifests (manifest_id, vehicle_id, departure_time, manifest_xml) VALUES (1, 'SLEIGH-01', '2025-12-22 06:00:00', '<manifest><passengers><passenger><name>Nia Grant</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Sofia Kim</name><ticket_class>overnight</ticket_class><engine_check>ignored</engine_check></passenger><passenger><name>Jonah Wolfe</name><ticket_class>standard</ticket_class><snack_inventory>ignored</snack_inventory></passenger></passengers></manifest>'), (2, 'SLEIGH-07', '2025-12-23 13:00:00', '<manifest><passengers><passenger><name>Ravi Patel</name><ticket_class>priority</ticket_class></passenger><passenger><name>Keiko Ito</name><ticket_class>standard</ticket_class></passenger><passenger><name>Anya Pavlov</name><ticket_class>standard</ticket_class></passenger><passenger><name>Nia Grant</name><ticket_class>priority</ticket_class><snack_inventory>ignored</snack_inventory></passenger><passenger><name>Carter Lewis</name><ticket_class>priority</ticket_class><engine_check>ignored</engine_check></passenger><passenger><name>Layla Brooks</name><ticket_class>standard</ticket_class><reindeer_mood>ignored</reindeer_mood></passenger></passengers></manifest>'), (3, 'FLIGHT-NP-9', '2025-12-22 18:00:00', '<manifest><passengers><passenger><name>Keiko Ito</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Jonah Wolfe</name><ticket_class>priority</ticket_class></passenger><passenger><name>Diego Ramos</name><ticket_class>overnight</ticket_class><snack_inventory>ignored</snack_inventory></passenger><passenger><name>Priya Das</name><ticket_class>priority</ticket_class></passenger><passenger><name>Layla Brooks</name><ticket_class>standard</ticket_class></passenger></passengers></manifest>'), (4, 'TRAIN-ICE-3', '2025-12-22 18:00:00', '<manifest><reindeer_mood>low</reindeer_mood><passengers><passenger><name>Isla Torres</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Ravi Patel</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Hiro Tanaka</name><ticket_class>standard</ticket_class></passenger><passenger><name>Priya Das</name><ticket_class>priority</ticket_class></passenger></passengers></manifest>'), (5, 'FLIGHT-NP-9', '2025-12-22 17:00:00', '<manifest><passengers><passenger><name>Nia Grant</name><ticket_class>priority</ticket_class></passenger><passenger><name>Mateo Cruz</name><ticket_class>standard</ticket_class><snowfall_inches>ignored</snowfall_inches></passenger></passengers></manifest>'), (6, 'CARGO-12', '2025-12-22 15:00:00', '<manifest><passengers><passenger><name>Carter Lewis</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Sofia Kim</name><ticket_class>priority</ticket_class></passenger><passenger><name>Hiro Tanaka</name><ticket_class>priority</ticket_class></passenger><passenger><name>Lucas Ford</name><ticket_class>priority</ticket_class><snack_inventory>ignored</snack_inventory></passenger><passenger><name>Nia Grant</name><ticket_class>overnight</ticket_class></passenger></passengers></manifest>'), (7, 'SLEIGH-01', '2025-12-22 11:00:00', '<manifest><snack_inventory>unknown</snack_inventory><passengers><passenger><name>Priya Das</name><ticket_class>standard</ticket_class></passenger><passenger><name>Diego Ramos</name><ticket_class>standard</ticket_class></passenger><passenger><name>Lucas Ford</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Carter Lewis</name><ticket_class>overnight</ticket_class><reindeer_mood>ignored</reindeer_mood></passenger><passenger><name>Hiro Tanaka</name><ticket_class>priority</ticket_class><weather_note>ignored</weather_note></passenger><passenger><name>Zara Sheikh</name><ticket_class>overnight</ticket_class></passenger></passengers></manifest>'), (8, 'CARGO-12', '2025-12-23 13:00:00', '<manifest><passengers><passenger><name>Jonah Wolfe</name><ticket_class>standard</ticket_class></passenger><passenger><name>Layla Brooks</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Leo Becker</name><ticket_class>overnight</ticket_class><weather_note>ignored</weather_note></passenger><passenger><name>Ravi Patel</name><ticket_class>standard</ticket_class></passenger><passenger><name>Sofia Kim</name><ticket_class>priority</ticket_class><snack_inventory>ignored</snack_inventory></passenger><passenger><name>Elena Morales</name><ticket_class>overnight</ticket_class></passenger></passengers></manifest>'), (9, 'SLEIGH-01', '2025-12-23 10:00:00', '<manifest><passengers><passenger><name>Sofia Kim</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Bianca Pereira</name><ticket_class>standard</ticket_class></passenger><passenger><name>Zara Sheikh</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Elena Morales</name><ticket_class>overnight</ticket_class><reindeer_mood>ignored</reindeer_mood></passenger><passenger><name>Hiro Tanaka</name><ticket_class>priority</ticket_class></passenger><passenger><name>Keiko Ito</name><ticket_class>priority</ticket_class><snack_inventory>ignored</snack_inventory></passenger></passengers></manifest>'), (10, 'SLEIGH-01', '2025-12-22 21:00:00', '<manifest><snowfall_inches>low</snowfall_inches><passengers><passenger><name>Nia Grant</name><ticket_class>standard</ticket_class></passenger><passenger><name>Ava Johnson</name><ticket_class>standard</ticket_class></passenger><passenger><name>Priya Das</name><ticket_class>priority</ticket_class></passenger><passenger><name>Mateo Cruz</name><ticket_class>standard</ticket_class></passenger><passenger><name>Bianca Pereira</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Leo Becker</name><ticket_class>overnight</ticket_class></passenger></passengers></manifest>'), (11, 'SLEIGH-07', '2025-12-23 10:00:00', '<manifest><passengers><passenger><name>Jonah Wolfe</name><ticket_class>priority</ticket_class></passenger><passenger><name>Bianca Pereira</name><ticket_class>priority</ticket_class></passenger><passenger><name>Anya Pavlov</name><ticket_class>standard</ticket_class></passenger></passengers></manifest>'), (12, 'SLEIGH-01', '2025-12-22 08:00:00', '<manifest><reindeer_mood>ok</reindeer_mood><passengers><passenger><name>Ravi Patel</name><ticket_class>priority</ticket_class><snack_inventory>ignored</snack_inventory></passenger><passenger><name>Bianca Pereira</name><ticket_class>priority</ticket_class></passenger><passenger><name>Keiko Ito</name><ticket_class>standard</ticket_class></passenger></passengers></manifest>'), (13, 'FLIGHT-NP-9', '2025-12-22 11:00:00', '<manifest><snowfall_inches>ok</snowfall_inches><passengers><passenger><name>Nia Grant</name><ticket_class>priority</ticket_class></passenger><passenger><name>Elena Morales</name><ticket_class>standard</ticket_class></passenger></passengers></manifest>'), (14, 'SLEIGH-01', '2025-12-22 14:00:00', '<manifest><engine_check>high</engine_check><passengers><passenger><name>Layla Brooks</name><ticket_class>standard</ticket_class><weather_note>ignored</weather_note></passenger><passenger><name>Anya Pavlov</name><ticket_class>overnight</ticket_class></passenger><passenger><name>Keiko Ito</name><ticket_class>standard</ticket_class></passenger><passenger><name>Zara Sheikh</name><ticket_class>standard</ticket_class><engine_check>ignored</engine_check></passenger><passenger><name>Sofia Kim</name><ticket_class>overnight</ticket_class></passenger></passengers></manifest>'), (15, 'FLIGHT-NP-9', '2025-12-22 14:00:00', '<manifest><snowfall_inches>ok</snowfall_inches><passengers><passenger><name>Priya Das</name><ticket_class>priority</ticket_class><snowfall_inches>ignored</snowfall_inches></passenger><passenger><name>Bianca Pereira</name><ticket_class>standard</ticket_class></passenger><passenger><name>Nia Grant</name><ticket_class>standard</ticket_class></passenger></passengers></manifest>'); ``` Just one table, with some wild XML. Let's see what do we want to do in the problem statement. ## Problem > Using the `travel_manifests` table, extract the passenger information from the XML data and produce a report that shows all of the departure times for "CARGO" vehicles that have more than 20 passengers booked. Include in the results: > > - The vehicle_id > - The departure_time > - The total number of passengers on that departure > - Order the results by departure_time. Ok, so we need the number of passengers in the records which are of type `CARGO` and have more than 20 passengers booked. Interesting! Let's look at one record. ``` sqlite> .schema CREATE TABLE travel_manifests ( manifest_id INT PRIMARY KEY, vehicle_id TEXT, departure_time TIMESTAMP, manifest_xml XML ); sqlite> SELECT * FROM travel_manifests WHERE id = 1; Parse error: no such column: id SELECT * FROM travel_manifests WHERE id = 1; error here ---^ sqlite> SELECT * FROM travel_manifests LIMIT 1; +-------------+------------+---------------------+--------------------------------------------------------------+ | manifest_id | vehicle_id | departure_time | manifest_xml | +-------------+------------+---------------------+--------------------------------------------------------------+ | 1 | SLEIGH-01 | 2025-12-22 06:00:00 | <manifest><passengers><passenger><name>Nia Grant</name><tick | | | | | et_class>overnight</ticket_class></passenger><passenger><nam | | | | | e>Sofia Kim</name><ticket_class>overnight</ticket_class><eng | | | | | ine_check>ignored</engine_check></passenger><passenger><name | | | | | >Jonah Wolfe</name><ticket_class>standard</ticket_class><sna | | | | | ck_inventory>ignored</snack_inventory></passenger></passenge | | | | | rs></manifest> | +-------------+------------+---------------------+--------------------------------------------------------------+ sqlite> ``` So, we have the following columns - `vehicle_id` which is I think related to filtering `CARGO` related vehicles only - `departure_time` which we just return as is - `manifest_xml`, oh! This is xml and it has passenger details. If we look carefully, we can see the xml looks like this: ```xml <manifest> <passengers> <passenger> <name>Nia Grant</name> <ticket_class>overnight</ticket_class> </passenger> <passenger> <name>Sofia Kim</name> <ticket_class>overnight</ticket_class> <engine_check>ignored</engine_check> </passenger> <passenger> <name>Jonah Wolfe</name> <ticket_class>standard</ticket_class> <snack_inventory>ignored</snack_inventory> </passenger> </passengers> </manifest> ``` We have `manifest` which has a property of `passengers` which is a list of `passenger` tags, inside of which, each element of `passenger` has its details like `name`, `ticket_class`, etc. We only want the count of `passengers`, how can we get that? The dirtiest way to do is to count the occurances of `<passenger>` or `</passenger>` in the xml string. We can do that with counting the full length of the xml string, and then dividing by the number of times the string can be replaced(which is the dirty part, there could be hidden `<passenger>` string somewhere that might break this logic, but if it is a valid xml, it works). We count the number of characters left after we replace the string `<passenger>` with empty string `''` so that we can get the difference of the total number of character and the number of characters occupied by the string `<passenger>`. This difference if we divide by the length of `<passenger>` will give us the count of the number of times the `<passenger>` string is present in the xml string. Let's take an example from the above. The length of the xml string is 374. ```sql SELECT LENGTH(manifest_xml) FROM travel_manifests LIMIT 1; ``` ``` sqlite> SELECT LENGTH(manifest_xml) FROM travel_manifests LIMIT 1; +----------------------+ | LENGTH(manifest_xml) | +----------------------+ | 374 | +----------------------+ sqlite> ``` Let's replace the occurances of `<passenger> with empty string in the `manifest_xml` string, like so: ```sql SELECT REPLACE(manifest_xml, '<passenger>', '') FROM travel_manifests LIMIT 1; ``` Now, we can see the string `<passenger>` is gone from the returned result set. We can try getting its length now. ```sql SELECT LENGTH( REPLACE(manifest_xml, '<passenger>', '') ) FROM travel_manifests LIMIT 1; ``` Now, it says `341` why? Because we removed (replaced with empty string) the occurences of `<passenger>`. Let's get the length of `'<passenger>'` string, which should be `11` right? Spell it `p-a-s-s-e-n-g-e-r` as `pass` + `enger` (4+5=9) and 2 for `<>` so 11. Sometimes I don't know how to do math, I use SQL. ```sql SELECT LENGTH('<passenger>'); ``` There it is `11`. Now, if you compute the difference of the actual length of XML with the removed parts of the `<passenger>` what do we get? ```sql SELECT LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')) FROM travel_manifests LIMIT 1; ``` WE got `33`, why would you ask because `11` times 3 is `33`. We found three instances of `<passenger>`, so we just need to divide by the length of `<passenger>` or hard code it as `11` doesn't matter. We would get the number of occurances of `<passenger>` which will give the number of passenger in the xml string. ```sql SELECT ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')) ) / LENGTH('<passenger>') FROM travel_manifests LIMIT 1; ``` Phew, its `3`! That was a lot for a simple stuff. But hey its fun! ``` sqlite> SELECT (manifest_xml) FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | manifest_xml | +--------------------------------------------------------------+ | <manifest><passengers><passenger><name>Nia Grant</name><tick | | et_class>overnight</ticket_class></passenger><passenger><nam | | e>Sofia Kim</name><ticket_class>overnight</ticket_class><eng | | ine_check>ignored</engine_check></passenger><passenger><name | | >Jonah Wolfe</name><ticket_class>standard</ticket_class><sna | | ck_inventory>ignored</snack_inventory></passenger></passenge | | rs></manifest> | +--------------------------------------------------------------+ sqlite> SELECT REPLACE(manifest_xml, '<passenger>', '') FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | REPLACE(manifest_xml, '<passenger>', '') | +--------------------------------------------------------------+ | <manifest><passengers><name>Nia Grant</name><ticket_class>ov | | ernight</ticket_class></passenger><name>Sofia Kim</name><tic | | ket_class>overnight</ticket_class><engine_check>ignored</eng | | ine_check></passenger><name>Jonah Wolfe</name><ticket_class> | | standard</ticket_class><snack_inventory>ignored</snack_inven | | tory></passenger></passengers></manifest> | +--------------------------------------------------------------+ sqlite> sqlite> SELECT LENGTH(REPLACE(manifest_xml, '<passenger>', '')) FROM travel_manifests LIMIT 1; +--------------------------------------------------+ | LENGTH(REPLACE(manifest_xml, '<passenger>', '')) | +--------------------------------------------------+ | 341 | +--------------------------------------------------+ sqlite> SELECT LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')) FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passen | +--------------------------------------------------------------+ | 33 | +--------------------------------------------------------------+ sqlite> SELECT LENGTH('<passenger>') FROM travel_manifests LIMIT 1; +-----------------------+ | LENGTH('<passenger>') | +-----------------------+ | 11 | +-----------------------+ sqlite> SELECT (LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')))/LENGTH('<passenger>') FROM travel_manifests LIMIT 1; +--------------------------------------------------------------+ | (LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passe | +--------------------------------------------------------------+ | 3 | +--------------------------------------------------------------+ sqlite> ``` Now, let construct the query to get the number of passengers. ```sql SELECT vehicle_id, departure_time, ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')) ) / LENGTH('<passenger>') AS passengers_in_manifest FROM travel_manifests WHERE vehicle_id LIKE 'CARGO-%'; ``` We need to wrap it in a CTE to grab and group by the vehicle_id I believe, as there are similar entries. Also we need to group the records with the same departure time, so that we can combine the number of passengers for that vehicle. ```sql WITH passenger_counts AS ( SELECT vehicle_id, departure_time, ( LENGTH(manifest_xml) - LENGTH(REPLACE(manifest_xml, '<passenger>', '')) ) / LENGTH('<passenger>') AS passengers_in_manifest FROM travel_manifests WHERE vehicle_id LIKE 'CARGO-%' ) SELECT vehicle_id, departure_time, SUM(passengers_in_manifest) AS total_passengers FROM passenger_counts GROUP BY vehicle_id, departure_time HAVING SUM(passengers_in_manifest) > 20 ORDER BY departure_time; ``` We count the number of passengers in the CTE and use it as a filter in the outer query as `SUM(passengers_in_manifest) > 20` which will give the right condition for us to get the result. We have to use `HAVING` as we need to do that after grouping the same `vehicle_id` and records across same `departure_time`. We also use the `vehicle_id LIKE 'CARGO-%'` in the CTE to filter it right at the inner query to avoid looping in all the queries for computing the number of passengers. That solves this problem. That's it from day 13 of Advent of SQL. There are other ways, but its the same parsing, We can use JOINs and stuff, but hey that was not the point of this. Anyways! See you tomorrow for day 14! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-11 Advent of SQL - Day 11, Behavior Score All right, this is day 11 from Advent of SQL. Let's pull in the data. No hiccups! Good to go. We just have one table toda Fri, 26 Dec 2025 00:00:00 UTC ## Advent of SQL - Day 11, Behavior Score All right, this is day 11 from Advent of SQL. Let's pull in the data. ```sql DROP TABLE IF EXISTS behavior_logs; CREATE TABLE behavior_logs ( id INT PRIMARY KEY, child_id INT, child_name TEXT, behavior_date DATE, score INT ); INSERT INTO behavior_logs (id, child_id, child_name, behavior_date, score) VALUES (1, 1, 'Emma D.', '2025-12-01', 5), (2, 1, 'Emma D.', '2025-12-02', 1), (3, 1, 'Emma D.', '2025-12-03', 3), (4, 1, 'Emma D.', '2025-12-04', 5), (5, 1, 'Emma D.', '2025-12-05', 2), (6, 1, 'Emma D.', '2025-12-06', 2), (7, 1, 'Emma D.', '2025-12-07', 3), (8, 1, 'Emma D.', '2025-12-08', 5), (9, 1, 'Emma D.', '2025-12-09', 4), (10, 1, 'Emma D.', '2025-12-10', 5), (11, 1, 'Emma D.', '2025-12-11', 5), (12, 1, 'Emma D.', '2025-12-12', -1), (13, 1, 'Emma D.', '2025-12-13', 1), (14, 1, 'Emma D.', '2025-12-14', 1), (15, 1, 'Emma D.', '2025-12-15', -1), (16, 1, 'Emma D.', '2025-12-16', 3), (17, 1, 'Emma D.', '2025-12-17', -2), (18, 1, 'Emma D.', '2025-12-18', 1), (19, 1, 'Emma D.', '2025-12-19', 1), (20, 1, 'Emma D.', '2025-12-20', -2), (21, 2, 'Ava X.', '2025-12-01', 0), (22, 2, 'Ava X.', '2025-12-02', -1), (23, 2, 'Ava X.', '2025-12-03', 4), (24, 2, 'Ava X.', '2025-12-04', 0), (25, 2, 'Ava X.', '2025-12-05', 2), (26, 2, 'Ava X.', '2025-12-06', 3), (27, 2, 'Ava X.', '2025-12-07', 5), (28, 2, 'Ava X.', '2025-12-08', 2), (29, 2, 'Ava X.', '2025-12-09', 1), (30, 2, 'Ava X.', '2025-12-10', 5), (31, 2, 'Ava X.', '2025-12-11', 2), (32, 2, 'Ava X.', '2025-12-12', 5), (33, 2, 'Ava X.', '2025-12-13', 5), (34, 2, 'Ava X.', '2025-12-14', 2), (35, 2, 'Ava X.', '2025-12-15', 0), (36, 2, 'Ava X.', '2025-12-16', 0), (37, 2, 'Ava X.', '2025-12-17', 5), (38, 2, 'Ava X.', '2025-12-18', 4), (39, 2, 'Ava X.', '2025-12-19', 5), (40, 2, 'Ava X.', '2025-12-20', 5), (181, 10, 'Ava C.', '2025-12-01', 3), (182, 10, 'Ava C.', '2025-12-02', 0), (183, 10, 'Ava C.', '2025-12-03', 3), (184, 10, 'Ava C.', '2025-12-04', 5), (185, 10, 'Ava C.', '2025-12-05', 5), (186, 10, 'Ava C.', '2025-12-06', 4), (187, 10, 'Ava C.', '2025-12-07', 1), (188, 10, 'Ava C.', '2025-12-08', 4), (189, 10, 'Ava C.', '2025-12-09', 5), (190, 10, 'Ava C.', '2025-12-10', 5), (191, 10, 'Ava C.', '2025-12-11', 5), (192, 10, 'Ava C.', '2025-12-12', 0), (193, 10, 'Ava C.', '2025-12-13', 0), (194, 10, 'Ava C.', '2025-12-14', 3), (195, 10, 'Ava C.', '2025-12-15', 1), (196, 10, 'Ava C.', '2025-12-16', 3), (197, 10, 'Ava C.', '2025-12-17', -1), (198, 10, 'Ava C.', '2025-12-18', 0), (199, 10, 'Ava C.', '2025-12-19', 5), (200, 10, 'Ava C.', '2025-12-20', 4), (241, 13, 'Ava R.', '2025-12-01', 3), (242, 13, 'Ava R.', '2025-12-02', 2), (243, 13, 'Ava R.', '2025-12-03', 2), (244, 13, 'Ava R.', '2025-12-04', 1), (245, 13, 'Ava R.', '2025-12-05', -1), (246, 13, 'Ava R.', '2025-12-06', -1), (247, 13, 'Ava R.', '2025-12-07', 2), (248, 13, 'Ava R.', '2025-12-08', 5), (249, 13, 'Ava R.', '2025-12-09', 0), (250, 13, 'Ava R.', '2025-12-10', 5), (251, 13, 'Ava R.', '2025-12-11', 2), (252, 13, 'Ava R.', '2025-12-12', -1), (253, 13, 'Ava R.', '2025-12-13', 2), (254, 13, 'Ava R.', '2025-12-14', 3), (255, 13, 'Ava R.', '2025-12-15', 2), (256, 13, 'Ava R.', '2025-12-16', -1), (257, 13, 'Ava R.', '2025-12-17', -2), (258, 13, 'Ava R.', '2025-12-18', -4), (259, 13, 'Ava R.', '2025-12-19', -3), (260, 13, 'Ava R.', '2025-12-20', 2), (1961, 99, 'Ava X.', '2025-12-01', 2), (1962, 99, 'Ava X.', '2025-12-02', -2), (1963, 99, 'Ava X.', '2025-12-03', -1), (1964, 99, 'Ava X.', '2025-12-04', -2), (1965, 99, 'Ava X.', '2025-12-05', 3), (1966, 99, 'Ava X.', '2025-12-06', -1), (1967, 99, 'Ava X.', '2025-12-07', 0), (1968, 99, 'Ava X.', '2025-12-08', 1), (1969, 99, 'Ava X.', '2025-12-09', 0), (1970, 99, 'Ava X.', '2025-12-10', 0), (1971, 99, 'Ava X.', '2025-12-11', 3), (1972, 99, 'Ava X.', '2025-12-12', 4), (1973, 99, 'Ava X.', '2025-12-13', 4), (1974, 99, 'Ava X.', '2025-12-14', 0), (1975, 99, 'Ava X.', '2025-12-15', 3), (1976, 99, 'Ava X.', '2025-12-16', -1), (1977, 99, 'Ava X.', '2025-12-17', -1), (1978, 99, 'Ava X.', '2025-12-18', 3), (1979, 99, 'Ava X.', '2025-12-19', 3), (1980, 99, 'Ava X.', '2025-12-20', -3); ``` No hiccups! Good to go. We just have one table today. ```sql SELECT * FROM behavior_logs; ``` ``` sqlite> .read day11-inserts.sql sqlite> .mode table sqlite> .schema CREATE TABLE behavior_logs ( id INT PRIMARY KEY, child_id INT, child_name TEXT, behavior_date DATE, score INT ); sqlite> SELECT * FROM behavior_logs LIMIT 20; +----+----------+------------+---------------+-------+ | id | child_id | child_name | behavior_date | score | +----+----------+------------+---------------+-------+ | 1 | 1 | Emma D. | 2025-12-01 | 5 | | 2 | 1 | Emma D. | 2025-12-02 | 1 | | 3 | 1 | Emma D. | 2025-12-03 | 3 | | 4 | 1 | Emma D. | 2025-12-04 | 5 | | 5 | 1 | Emma D. | 2025-12-05 | 2 | | 6 | 1 | Emma D. | 2025-12-06 | 2 | | 7 | 1 | Emma D. | 2025-12-07 | 3 | | 8 | 1 | Emma D. | 2025-12-08 | 5 | | 9 | 1 | Emma D. | 2025-12-09 | 4 | | 10 | 1 | Emma D. | 2025-12-10 | 5 | | 11 | 1 | Emma D. | 2025-12-11 | 5 | | 12 | 1 | Emma D. | 2025-12-12 | -1 | | 13 | 1 | Emma D. | 2025-12-13 | 1 | | 14 | 1 | Emma D. | 2025-12-14 | 1 | | 15 | 1 | Emma D. | 2025-12-15 | -1 | | 16 | 1 | Emma D. | 2025-12-16 | 3 | | 17 | 1 | Emma D. | 2025-12-17 | -2 | | 18 | 1 | Emma D. | 2025-12-18 | 1 | | 19 | 1 | Emma D. | 2025-12-19 | 1 | | 20 | 1 | Emma D. | 2025-12-20 | -2 | +----+----------+------------+---------------+-------+ sqlite> SELECT * FROM behavior_logs WHERE child_name LIKE 'Ava'; sqlite> SELECT * FROM behavior_logs WHERE child_name LIKE 'Ava%'; +------+----------+------------+---------------+-------+ | id | child_id | child_name | behavior_date | score | +------+----------+------------+---------------+-------+ | 21 | 2 | Ava X. | 2025-12-01 | 0 | | 22 | 2 | Ava X. | 2025-12-02 | -1 | | 23 | 2 | Ava X. | 2025-12-03 | 4 | | 24 | 2 | Ava X. | 2025-12-04 | 0 | | 25 | 2 | Ava X. | 2025-12-05 | 2 | | 26 | 2 | Ava X. | 2025-12-06 | 3 | | 27 | 2 | Ava X. | 2025-12-07 | 5 | | 28 | 2 | Ava X. | 2025-12-08 | 2 | | 29 | 2 | Ava X. | 2025-12-09 | 1 | | 30 | 2 | Ava X. | 2025-12-10 | 5 | | 31 | 2 | Ava X. | 2025-12-11 | 2 | | 32 | 2 | Ava X. | 2025-12-12 | 5 | | 33 | 2 | Ava X. | 2025-12-13 | 5 | | 34 | 2 | Ava X. | 2025-12-14 | 2 | | 35 | 2 | Ava X. | 2025-12-15 | 0 | | 36 | 2 | Ava X. | 2025-12-16 | 0 | | 37 | 2 | Ava X. | 2025-12-17 | 5 | | 38 | 2 | Ava X. | 2025-12-18 | 4 | | 39 | 2 | Ava X. | 2025-12-19 | 5 | | 40 | 2 | Ava X. | 2025-12-20 | 5 | | 181 | 10 | Ava C. | 2025-12-01 | 3 | | 182 | 10 | Ava C. | 2025-12-02 | 0 | | 183 | 10 | Ava C. | 2025-12-03 | 3 | | 184 | 10 | Ava C. | 2025-12-04 | 5 | | 185 | 10 | Ava C. | 2025-12-05 | 5 | | 186 | 10 | Ava C. | 2025-12-06 | 4 | | 187 | 10 | Ava C. | 2025-12-07 | 1 | | 188 | 10 | Ava C. | 2025-12-08 | 4 | | 189 | 10 | Ava C. | 2025-12-09 | 5 | | 190 | 10 | Ava C. | 2025-12-10 | 5 | | 191 | 10 | Ava C. | 2025-12-11 | 5 | | 192 | 10 | Ava C. | 2025-12-12 | 0 | | 193 | 10 | Ava C. | 2025-12-13 | 0 | | 194 | 10 | Ava C. | 2025-12-14 | 3 | | 195 | 10 | Ava C. | 2025-12-15 | 1 | | 196 | 10 | Ava C. | 2025-12-16 | 3 | | 197 | 10 | Ava C. | 2025-12-17 | -1 | | 198 | 10 | Ava C. | 2025-12-18 | 0 | | 199 | 10 | Ava C. | 2025-12-19 | 5 | | 200 | 10 | Ava C. | 2025-12-20 | 4 | | 241 | 13 | Ava R. | 2025-12-01 | 3 | | 242 | 13 | Ava R. | 2025-12-02 | 2 | | 243 | 13 | Ava R. | 2025-12-03 | 2 | | 244 | 13 | Ava R. | 2025-12-04 | 1 | | 245 | 13 | Ava R. | 2025-12-05 | -1 | | 246 | 13 | Ava R. | 2025-12-06 | -1 | | 247 | 13 | Ava R. | 2025-12-07 | 2 | | 248 | 13 | Ava R. | 2025-12-08 | 5 | | 249 | 13 | Ava R. | 2025-12-09 | 0 | | 250 | 13 | Ava R. | 2025-12-10 | 5 | | 251 | 13 | Ava R. | 2025-12-11 | 2 | | 252 | 13 | Ava R. | 2025-12-12 | -1 | | 253 | 13 | Ava R. | 2025-12-13 | 2 | | 254 | 13 | Ava R. | 2025-12-14 | 3 | | 255 | 13 | Ava R. | 2025-12-15 | 2 | | 256 | 13 | Ava R. | 2025-12-16 | -1 | | 257 | 13 | Ava R. | 2025-12-17 | -2 | | 258 | 13 | Ava R. | 2025-12-18 | -4 | | 259 | 13 | Ava R. | 2025-12-19 | -3 | | 260 | 13 | Ava R. | 2025-12-20 | 2 | | 1961 | 99 | Ava X. | 2025-12-01 | 2 | | 1962 | 99 | Ava X. | 2025-12-02 | -2 | | 1963 | 99 | Ava X. | 2025-12-03 | -1 | | 1964 | 99 | Ava X. | 2025-12-04 | -2 | | 1965 | 99 | Ava X. | 2025-12-05 | 3 | | 1966 | 99 | Ava X. | 2025-12-06 | -1 | | 1967 | 99 | Ava X. | 2025-12-07 | 0 | | 1968 | 99 | Ava X. | 2025-12-08 | 1 | | 1969 | 99 | Ava X. | 2025-12-09 | 0 | | 1970 | 99 | Ava X. | 2025-12-10 | 0 | | 1971 | 99 | Ava X. | 2025-12-11 | 3 | | 1972 | 99 | Ava X. | 2025-12-12 | 4 | | 1973 | 99 | Ava X. | 2025-12-13 | 4 | | 1974 | 99 | Ava X. | 2025-12-14 | 0 | | 1975 | 99 | Ava X. | 2025-12-15 | 3 | | 1976 | 99 | Ava X. | 2025-12-16 | -1 | | 1977 | 99 | Ava X. | 2025-12-17 | -1 | | 1978 | 99 | Ava X. | 2025-12-18 | 3 | | 1979 | 99 | Ava X. | 2025-12-19 | 3 | | 1980 | 99 | Ava X. | 2025-12-20 | -3 | +------+----------+------------+---------------+-------+ sqlite> ``` Let's get to the problem of day 11 ## Problem > Calculate the 7-day rolling average behavior score for each child. Identify any child whose rolling average drops below 0. For those children with a rolling average below 0, return the `child_id`, `child_name`, `behavior_date` (this will be the latest date in the 7-day rolling average), and the calculated 7-day rolling average. Only include results with a `behavior_date` of `December 7, 2025` or later, ensuring that each rolling average is based on a full 7 days of data. > > Order the results by `behavior_date` and then `child_name`. So, we need to do what? - Group by child_id - Compute the rolling average for the past 7 days (so if there is a data for 20 days from 1st to 20th December, we'll only consider the average from 14th to 20th December) - Order by behavior_date and child_name. ### Using Simple Join We have to first grab the rolling average which is only for the past 7 days per child. So, to do that, we can self join the `behavior_log` table on the condition that the behavior date (from right table) is between the current behavior date (from left table) and 6 days past that for each child. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id; ``` We basically joined the table `behavior_logs` with itself, i.e. the right and left tables are the same. So, we join on the condition of same child_id, and then look for dates between the past 6 days and the current behavior date (we included the 7th day in the behavior date, so hence subtracting 6 days from that date). We group by `child_id`, so that we get a single row for each `child_id`. We compute the `AVG(score)` to get the average score for each day in the range of the past 7 days. You see the issue, we need to compute the average for each week, and not just the last, it will give us the last most score for a child. We need to also group by the behavior date so as to keep week unique and have multiple entries for a child. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date; ``` Now, we have grouped by `behavior_date` so that each child can have multiple entries for the weeks that we have logs for. Now we can further filter it. We also need to filter when the `rolling_average` is less than 0, that is the child had a bad week overall. So that will be in a `HAVING` condition and not a `WHERE` condition since `AVG` is a aggregate function, we can't reference the `rolling_average` in the where clause, it won't be defined there yet. So, we use `HAVING` to filter `rolling_average` as less than `0`. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0; ``` We are now down to the rows that only have `bad` weeks for the child in various weeks. There is one catch however, we can't use week periods before `7th December` because we don't have enough data to compute the rolling average for the 7 day score. Hence we only include the records having logs after `7th December`. This again would come in the `HAVING` clause as we are deciding the final behavior date from the week and not the individual logs from the table. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0 AND behavior_logs.behavior_date >= '2025-12-07' ``` Now, the final piece is the order. We need to order it by the `behavior_date` and the `child_name` as requested. ```sql SELECT behavior_logs.child_id, behavior_logs.child_name, behavior_logs.behavior_date, AVG(week_logs.score) AS rolling_avg FROM behavior_logs JOIN behavior_logs week_logs ON behavior_logs.child_id = week_logs.child_id AND week_logs.behavior_date BETWEEN DATE(behavior_logs.behavior_date, '-6 days') AND behavior_logs.behavior_date GROUP BY behavior_logs.child_id, behavior_logs.behavior_date HAVING rolling_avg < 0 AND behavior_logs.behavior_date >= '2025-12-07' ORDER BY behavior_logs.behavior_date, behavior_logs.child_name; ``` This is a simple solution. Easy to understand, and explain, not quite short and crisp. We can even do this in a sub query without a JOIN like so ### Using Sub-query We take the conditions from the `JOIN` and replace that in a sub-query like so. ```sql SELECT current_logs.child_id, current_logs.child_name, current_logs.behavior_date, ( SELECT AVG(past_logs.score) FROM behavior_logs past_logs WHERE past_logs.child_id = current_logs.child_id AND past_logs.behavior_date BETWEEN date(current_logs.behavior_date, '-6 days') AND current_logs.behavior_date ) AS rolling_avg FROM behavior_logs current_logs WHERE current_logs.behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY current_logs.behavior_date, current_logs.child_name; ``` Since its a subquery, we don't need to group by or add a having clause to filter the rolling average and the behavior date. ### Using Sub-query and Window Function We can take the above query and instead of join, we can write a sub-query to compute the rolling average using window function. ```sql SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ``` We use the `AVG(score) OVER ()` this is a partition. We partition or create a window for each child and we order by the behavior date and then create a sliding window for the past 7 days. The `ROWS BETWEEN 6 PRECEDING AND CURRENT ROW` defines a 7-row sliding window for each child. Every row gets its own window, and each window is separate per child because of PARTITION BY `child_id`. So we get a full per day rolling average for each child with this query. This had around 2400 rows. Now, we need filter it down to the only rows where the rolling average is less than `0` and we don't include the average for days before the `7th December` as we don't have enough days before that to compute the 7-day rolling average. But this is a query in itself we can't reference the `rolling_average` in the where clause as its not available there. We haven't grouped by anything explicitly so we can't use `HAVING`, we need to wrap it in a sub query to get the rolling average as well as the behavior date condition. ```sql SELECT child_id, child_name, behavior_date, rolling_avg FROM ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0; ``` This will filter down the rows. Now, we also need to order by the `behavior_date` and then `child_name`. ```sql SELECT child_id, child_name, behavior_date, rolling_avg FROM ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY behavior_date, child_name; ``` So, that is again the same stuff just with window function. We can even do this with that subquery wrapped in a CTE (it just looks and reads good, nothing different really, other than you can use the CTE in the same query multiple times, here we don't need to do it) ### Using CTE and Window Function We just take the above sub-query and wrap it in a CTE. ```sql WITH rolling AS ( SELECT child_id, child_name, behavior_date, AVG(score) OVER ( PARTITION BY child_id ORDER BY behavior_date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW ) AS rolling_avg FROM behavior_logs ) SELECT child_id, child_name, behavior_date, rolling_avg FROM rolling WHERE behavior_date >= '2025-12-07' AND rolling_avg < 0 ORDER BY behavior_date, child_name; ``` The rest remains the same, we just reference the `rolling` as the CTE and grab the necessary details in the query using that CTE. That should be it for the Day 11! Some cool CTE, Window function and pratical use case for computing rolling averages, loved it! On to the day 12! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-10 Advent of SQL, Day 10 - Misdelivered Presents It's already day 10? We just need 5 more days now! Whoa! that flew by swiftly. Let's pull in the data. This is the Thu, 25 Dec 2025 00:00:00 UTC ## Advent of SQL, Day 10 - Misdelivered Presents It's already day 10? We just need 5 more days now! Whoa! that flew by swiftly. Let's pull in the data. This is the SQL for day 10 in SQLite. ```sql DROP TABLE IF EXISTS misdelivered_presents; DROP TABLE IF EXISTS deliveries; CREATE TABLE deliveries ( id INT PRIMARY KEY, child_name TEXT, delivery_location TEXT, gift_name TEXT, scheduled_at TIMESTAMP ); CREATE TABLE misdelivered_presents ( id INT PRIMARY KEY, child_name TEXT, delivery_location TEXT, gift_name TEXT, scheduled_at TIMESTAMP, flagged_at TIMESTAMP, reason TEXT ); INSERT INTO deliveries (id, child_name, delivery_location, gift_name, scheduled_at) VALUES (1, 'Omar Q.', '45 Maple Street', 'storybook collection', '2025-12-24 21:09:00'), (2, 'Sofia K.', '77 Snowflake Road', 'plush reindeer', '2025-12-24 18:35:00'), (3, 'Mila N.', 'The Vibes', 'storybook collection', '2025-12-24 21:09:00'), (4, 'Elias M.', 'Frost Hollow Cabin', 'board game', '2025-12-24 20:31:00'), (5, 'Ravi P.', '45 Maple Street', 'wooden train set', '2025-12-24 18:23:00'), (6, 'Jonah W.', '77 Snowflake Road', 'plush reindeer', '2025-12-24 20:34:00'), (7, 'Ava J.', '123 Evergreen Lane', 'board game', '2025-12-24 21:03:00'), (8, 'Omar Q.', '77 Snowflake Road', 'board game', '2025-12-24 18:56:00'), (9, 'Nia G.', 'Frost Hollow Cabin', 'teddy bear', '2025-12-24 21:27:00'), (10, 'Zara S.', 'North Pole Annex', 'wooden train set', '2025-12-24 20:58:00'), (11, 'Ravi P.', 'Frost Hollow Cabin', 'puzzle box', '2025-12-24 18:39:00'), (12, 'Jonah W.', '123 Evergreen Lane', 'puzzle box', '2025-12-24 18:23:00'), (13, 'Ravi P.', 'North Pole Annex', 'storybook collection', '2025-12-24 21:36:00'), (14, 'Lena F.', 'North Pole Annex', 'teddy bear', '2025-12-24 21:26:00'), (15, 'Ava J.', 'North Pole Annex', 'snow globe', '2025-12-24 18:31:00'), (16, 'Elias M.', 'Frost Hollow Cabin', 'robot toy', '2025-12-24 20:21:00'), (17, 'Sofia K.', 'Frost Hollow Cabin', 'teddy bear', '2025-12-24 20:27:00'), (18, 'Jonah W.', '77 Snowflake Road', 'storybook collection', '2025-12-24 20:49:00'), (19, 'Jonah W.', 'Frost Hollow Cabin', 'art supplies', '2025-12-24 21:38:00'), (20, 'Jonah W.', '123 Evergreen Lane', 'storybook collection', '2025-12-24 19:11:00'); INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason) VALUES (601, 'Priya D.', 'The Vibes', 'plush reindeer', '2025-12-24 14:00:00', '2025-12-24 14:05:00', 'Invalid delivery location'), (602, 'Lena F.', 'Abandoned Lighthouse', 'board game', '2025-12-22 06:00:00', '2025-12-22 06:05:00', 'Invalid delivery location'), (603, 'Caleb O.', 'Drifting Igloo', 'board game', '2025-12-24 06:00:00', '2025-12-24 06:05:00', 'Invalid delivery location'), (604, 'Mateo C.', 'The Vibes', 'art supplies', '2025-12-22 04:00:00', '2025-12-22 04:05:00', 'Invalid delivery location'), (605, 'Hiro T.', 'The Vibes', 'robot toy', '2025-12-24 08:00:00', '2025-12-24 08:05:00', 'Invalid delivery location'), (606, 'Priya D.', 'Volcano Rim', 'puzzle box', '2025-12-22 08:00:00', '2025-12-22 08:05:00', 'Invalid delivery location'), (607, 'Nia G.', 'Abandoned Lighthouse', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (608, 'Elias M.', 'Drifting Igloo', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (609, 'Ravi P.', 'Volcano Rim', 'board game', '2025-12-24 02:00:00', '2025-12-24 02:05:00', 'Invalid delivery location'), (610, 'Hiro T.', 'Abandoned Lighthouse', 'science kit', '2025-12-23 20:00:00', '2025-12-23 20:05:00', 'Invalid delivery location'), (611, 'Priya D.', 'Drifting Igloo', 'puzzle box', '2025-12-22 21:00:00', '2025-12-22 21:05:00', 'Invalid delivery location'), (612, 'Hiro T.', 'Volcano Rim', 'art supplies', '2025-12-23 09:00:00', '2025-12-23 09:05:00', 'Invalid delivery location'), (613, 'Jonah W.', 'Abandoned Lighthouse', 'board game', '2025-12-24 01:00:00', '2025-12-24 01:05:00', 'Invalid delivery location'), (614, 'Omar Q.', 'Volcano Rim', 'art supplies', '2025-12-22 01:00:00', '2025-12-22 01:05:00', 'Invalid delivery location'), (615, 'Omar Q.', 'Drifting Igloo', 'science kit', '2025-12-23 20:00:00', '2025-12-23 20:05:00', 'Invalid delivery location'), (616, 'Omar Q.', 'Abandoned Lighthouse', 'teddy bear', '2025-12-24 12:00:00', '2025-12-24 12:05:00', 'Invalid delivery location'), (617, 'Zara S.', 'Volcano Rim', 'wooden train set', '2025-12-24 12:00:00', '2025-12-24 12:05:00', 'Invalid delivery location'), (618, 'Omar Q.', 'Abandoned Lighthouse', 'teddy bear', '2025-12-23 15:00:00', '2025-12-23 15:05:00', 'Invalid delivery location'), (619, 'Caleb O.', 'The Vibes', 'teddy bear', '2025-12-24 14:00:00', '2025-12-24 14:05:00', 'Invalid delivery location'), (620, 'Nia G.', 'Abandoned Lighthouse', 'board game', '2025-12-23 03:00:00', '2025-12-23 03:05:00', 'Invalid delivery location'); ``` ```sql SELECT * FROM deliveries; SELECT * FROM misdelivered_presents; ``` We have two tables. Almost the same with critical logical distinction among them and one extra column. Let's see the problem to check it. ## Problem > Clean-up the deliveries table to remove any records where the delivery_location is 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'. > > Move those records to the misdelivered_presents with all the same columns as deliveries plus a flagged_at column with the current time and a reason column with "Invalid delivery location" listed as the reason for each moved record. > > Make sure your final step shows the misdelivered_presents records that you just moved (i.e. don't include any existing records from the misdelivered_presents table). Ok, this looks some easy problem. - SELECT some data - INSERT that data into other table - DELETE that data from the original table - SELECT the newly inserted data in the other table Right? Unless! > Santa turned to you. > > “I don’t want this done in five steps,” he said. “And I don’t want any re-selecting. Move the problem presents out of the delivery system, log them in the vault, and show me exactly what you moved.” Ouch Santa! Don't be lazy! Be smart he says! Huhh! Ok, at least let's check both the tables, how many rows they have, and the rows that we want to move around. ```sql SELECT COUNT(*) AS delivery_count FROM deliveries; SELECT COUNT(*) AS misdelivered_present_count FROM misdelivered_presents; SELECT COUNT(*) AS misdelivered_deliveries_count FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ); SELECT COUNT(*) AS misdelivered_present_count FROM misdelivered_presents WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ); ``` ``` sqlite> SELECT COUNT(*) FROM deliveries; +----------+ | COUNT(*) | +----------+ | 600 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents; +----------+ | COUNT(*) | +----------+ | 50 | +----------+ sqlite> SELECT COUNT(*) FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); +----------+ | COUNT(*) | +----------+ | 103 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); +----------+ | COUNT(*) | +----------+ | 50 | +----------+ sqlite> ``` OK, so we need to delete the `103` records from the `deliveries` table and move them into the `misdelivered_presents` at the same time. How are we going to do that in SQLite. We can try - DELETE FROM deliveries with a SELECT from deliveries - Then INSERT the deleted data into misdelivered_presents Could that work? Let's see ```sql WITH moved AS ( DELETE FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes' ) RETURNING id, child_name, delivery_location, gift_name, scheduled_at, datetime('now') AS flagged_at, 'Invalid delivery location' AS reason ) SELECT * FROM moved; ``` ``` ``` Ops! Can't generate a CTE with delete in it. That's nasty. Thought that could shove it in the insert into the misdelivered table. But it should be other way then? Insert first and then use the data to delete? ```sql WITH inserted_data AS ( INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at) SELECT id, child_name, delivery_location, gift_name, scheduled_at FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); ``` ``` sqlite> WITH inserted_data AS ( INSERT INTO misdelivered_presents (id, child_name, delivery_location, gift_name, scheduled_at) SELECT id, child_name, delivery_location, gift_name, scheduled_at FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes'); Parse error: near "INSERT": syntax error rt the selected rows into misdelivered_presents INSERT INTO misdelivered_pre error here ---^ sqlite> ``` Tried it, didn't work Quoting the documentation here. > Common Table Expressions or CTEs act like temporary views that exist only for the duration of a single SQL statement. There are two kinds of common table expressions: "ordinary" and "recursive". Ordinary common table expressions are helpful for making queries easier to understand by factoring subqueries out of the main SQL statement. Recursive common table expressions provide the ability to do hierarchical or recursive queries of trees and graphs, a capability that is not otherwise available in the SQL language. I think it doesn't support CTEs with delete or insert! Sigh! Now? BEGIN COMMIT? Atomic Transactions? Yeah! Santa wanted it in one go right? That's not possible in SQLite but at least everything will happen or nothing will. ```sql BEGIN; WITH misdelivered_deliveries AS ( SELECT * FROM deliveries WHERE delivery_location IN ( 'Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ) INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) SELECT id, child_name, delivery_location, gift_name, scheduled_at, DATETIME('now'), 'Invalid delivery location' FROM misdelivered_deliveries RETURNING *; DELETE FROM deliveries WHERE id IN ( SELECT id FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') ); COMMIT; ``` So here, we defined the CTE that selects data from `deliveries` and uses it to insert into the `misdelivered_deliveries` table. Then a separate query to delete that data from the `deliveries` table. Yeah! I mean I don't think there could be other way to it! We might or can use triggers to insert into one table when inserted in one table. But I think that is too much of a farfetched solution. We might create a trigger and instantly delete it as it could populate unwanted data if kept in the database. ### Trigger to insert when deleted We can create a `TRIGGER` to insert into `misdelivered_presents` when something is deleted from `deliveries` table. We will separately have to delete the records from the `deliveries` table. But the insert will happen automatically after the deletion. Opening a fresh instance of the database! ```sql CREATE TRIGGER move_misdelivered_presents BEFORE DELETE ON deliveries FOR EACH ROW WHEN OLD.delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') BEGIN INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) VALUES ( OLD.id, OLD.child_name, OLD.delivery_location, OLD.gift_name, OLD.scheduled_at, DATETIME('now'), 'Invalid delivery location' ); END; ``` This will create the trigger to insert the row into `misdelivered_presents` table when deleted from the `deliveries` table. ```sql DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') RETURNING *; ``` Ops! We are returning from the DELETE statement which is wrong as the problem stated to select from the `misdelivered_presents` table. > Make sure your final step shows the misdelivered_presents records that you just moved (i.e. don't include any existing records from the misdelivered_presents table). This is invalid then! Though its not technically atomic. It happens before the delete, so it can mess up things. ``` sqlite> .read day10-inserts.sql sqlite> .mode table sqlite> CREATE TRIGGER move_misdelivered_presents BEFORE DELETE ON deliveries FOR EACH ROW WHEN OLD.delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') BEGIN INSERT INTO misdelivered_presents ( id, child_name, delivery_location, gift_name, scheduled_at, flagged_at, reason ) VALUES ( OLD.id, OLD.child_name, OLD.delivery_location, OLD.gift_name, OLD.scheduled_at, DATETIME('now'), 'Invalid delivery location' ); END; sqlite> DELETE FROM deliveries WHERE delivery_location IN ('Volcano Rim', 'Drifting Igloo', 'Abandoned Lighthouse', 'The Vibes') RETURNING *; +-----+------------+----------------------+----------------------+---------------------+ | id | child_name | delivery_location | gift_name | scheduled_at | +-----+------------+----------------------+----------------------+---------------------+ | 3 | Mila N. | The Vibes | storybook collection | 2025-12-24 21:09:00 | | 22 | Lena F. | Abandoned Lighthouse | plush reindeer | 2025-12-24 19:08:00 | | 23 | Mila N. | Abandoned Lighthouse | storybook collection | 2025-12-24 20:42:00 | | 29 | Mateo C. | Volcano Rim | plush reindeer | 2025-12-24 21:44:00 | | 31 | Nia G. | Drifting Igloo | robot toy | 2025-12-24 19:57:00 | ... ... | 582 | Zara S. | The Vibes | teddy bear | 2025-12-24 21:20:00 | | 585 | Layla B. | Abandoned Lighthouse | wooden train set | 2025-12-24 18:39:00 | | 587 | Nia G. | Volcano Rim | storybook collection | 2025-12-24 18:35:00 | | 596 | Omar Q. | Abandoned Lighthouse | puzzle box | 2025-12-24 19:28:00 | +-----+------------+----------------------+----------------------+---------------------+ sqlite> DROP TRIGGER move_misdelivered_presents; sqlite> SELECT COUNT(*) FROM deliveries; +----------+ | COUNT(*) | +----------+ | 497 | +----------+ sqlite> SELECT COUNT(*) FROM misdelivered_presents; +----------+ | COUNT(*) | +----------+ | 153 | +----------+ sqlite> ``` So, it gets the result but I don't think the `TRIGGER` is a solution to this. In SQLite, Atomic transaction using begin and end is the only thing to go for right? Some one prove Santa that it can't be done in SQLite in one query? Please! That's it from day 10, I have spent enough time on this, banging my head on the sqlite shell! Off to day 11 tomorrow! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-9 Advent of SQL, Day 9 - Evergreen Market Orders We are on day 9 of advent of SQL, and I feel good so far. Let's see what we learn today? Let's get the inserts fo Wed, 24 Dec 2025 00:00:00 UTC ## Advent of SQL, Day 9 - Evergreen Market Orders We are on day 9 of advent of SQL, and I feel good so far. Let's see what we learn today? Let's get the inserts for the day. ``` sqlite> .read day9-inserts.sql sqlite> .schema CREATE TABLE orders ( id INT PRIMARY KEY, customer_id INT, created_at TIMESTAMP, order_data JSONB ); sqlite> .mode table sqlite> SELECT * FROM orders limit 10; +----+-------------+---------------------+--------------------------------------------------------------+ | id | customer_id | created_at | order_data | +----+-------------+---------------------+--------------------------------------------------------------+ | 1 | 1 | 2025-11-21 13:08:22 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | | | | | e}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 2 | 1 | 2025-11-21 18:42:58 | {"shipping": {"method": "overnight"}, "risk": {"flag": "high | | | | | "}, "gift": {"wrapped": false}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 3 | 1 | 2025-11-21 21:01:46 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | | | | | se}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 4 | 1 | 2025-11-24 13:17:27 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | | | | | e}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 5 | 1 | 2025-11-24 21:09:46 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": fa | | | | | lse}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 6 | 1 | 2025-11-25 07:24:55 | {"shipping": {"method": "standard"}, "risk": {"flag": "mediu | | | | | m"}, "gift": {"wrapped": true}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 7 | 1 | 2025-11-25 17:42:36 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | | | | | se}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 8 | 1 | 2025-11-27 02:34:24 | {"shipping": {"method": "express"}, "gift": {"wrapped": true | | | | | }} | +----+-------------+---------------------+--------------------------------------------------------------+ | 9 | 1 | 2025-11-30 22:43:54 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": tr | | | | | ue}} | +----+-------------+---------------------+--------------------------------------------------------------+ | 10 | 1 | 2025-12-01 04:03:33 | {"shipping": {"method": "express"}, "risk": {"flag": "medium | | | | | "}, "gift": {"wrapped": false}} | +----+-------------+---------------------+--------------------------------------------------------------+ sqlite> ``` Looks like we will deal with JSON today, seems exciting. I haven't dealt with JSON in SQLite yet, today will break it. Let's get some sample inserts for you to play in the browser. LIMITING TO 20, there are more than 400 rows! ```sql DROP TABLE IF EXISTS orders; CREATE TABLE orders ( id INT PRIMARY KEY, customer_id INT, created_at TIMESTAMP, order_data JSONB ); INSERT INTO orders (id, customer_id, created_at, order_data) VALUES (1, 1, '2025-11-21 13:08:22', '{"shipping": {"method": "standard"}, "gift": {"wrapped": true}}'), (2, 1, '2025-11-21 18:42:58', '{"shipping": {"method": "overnight"}, "risk": {"flag": "high"}, "gift": {"wrapped": false}}'), (3, 1, '2025-11-21 21:01:46', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (4, 1, '2025-11-24 13:17:27', '{"shipping": {"method": "standard"}, "gift": {"wrapped": true}}'), (5, 1, '2025-11-24 21:09:46', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": false}}'), (6, 1, '2025-11-25 07:24:55', '{"shipping": {"method": "standard"}, "risk": {"flag": "medium"}, "gift": {"wrapped": true}}'), (7, 1, '2025-11-25 17:42:36', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (8, 1, '2025-11-27 02:34:24', '{"shipping": {"method": "express"}, "gift": {"wrapped": true}}'), (9, 1, '2025-11-30 22:43:54', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": true}}'), (10, 1, '2025-12-01 04:03:33', '{"shipping": {"method": "express"}, "risk": {"flag": "medium"}, "gift": {"wrapped": false}}'), (11, 1, '2025-12-02 05:19:10', '{"shipping": {"method": "overnight"}, "risk": {"flag": "low"}, "gift": {"wrapped": true}}'), (12, 1, '2025-12-03 16:25:56', '{"shipping": {"method": "overnight"}, "risk": {"flag": "medium"}, "gift": {"wrapped": true}}'), (13, 1, '2025-12-10 19:34:28', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (14, 1, '2025-12-16 19:23:53', '{"shipping": {"method": "express"}, "gift": {"wrapped": false}}'), (15, 2, '2025-11-23 19:11:23', '{"shipping": {"method": "overnight"}, "gift": {"wrapped": true}}'), (16, 2, '2025-11-28 15:23:27', '{"shipping": {"method": "express"}, "gift": {"wrapped": true}}'), (17, 2, '2025-11-30 12:05:36', '{"shipping": {"method": "overnight"}, "risk": {"flag": "low"}, "gift": {"wrapped": true}}'), (18, 2, '2025-12-03 07:03:06', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'), (19, 2, '2025-12-07 13:55:13', '{"shipping": {"method": "express"}, "risk": {"flag": "high"}, "gift": {"wrapped": false}}'), (20, 2, '2025-12-08 07:17:31', '{"shipping": {"method": "standard"}, "gift": {"wrapped": false}}'); ``` ```sql SELECT * FROM orders; ``` Let's get to the problem now. ## Problem > Build a report using the orders table that shows the latest order for each customer, along with their requested shipping method, gift wrap choice (as true or false), and the risk flag in separate columns. > > Order the report by the most recent order first so Evergreen Market can reach out to them ASAP. Ok, so we need for each customer the latest order with the following details: - shipping method - gift wrap choice - risk flag These all I think are in the same column as a JSON string or blob. We need to extract those out from the column. Let's first check the `orders` table. It has a few columns: - id - customer_id - created_at - order_data We do require the `order_data` as that is the column that has JSON. Also the problem said to give the most recent order, so we need to order by `created_at` in a reverse way, the latest first. Also we need it per customer, so we need to group by `customer_id`. Let's see how to get the data inside JSON in SQLite. ### JSON Extract Well we have [json_extract](https://sqlite.org/json1.html#jex) which can give us the value of the key from the given json data string. The function takes in the column name from the table which would be the column containing the json data and then the second parameter is the path to the key, in this case if we want to get the `method` from the `shipping` key, we can use `$.shipping.method` which means from the root `$` get the `shipping` key, and inside that (`shipping` key) give the value inside the `method` key. If the path is not present, in our case the `risk` key is very rarely present in the actual original json data, the function skips the further key lookup, it returns `NULL`. ```sql SELECT json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; ``` ``` sqlite> select json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; +------------------------------------------------------+ | json_extract(orders.order_data, '$.shipping.method') | +------------------------------------------------------+ | standard | | overnight | | standard | | standard | | overnight | +------------------------------------------------------+ sqlite> select *, json_extract(orders.order_data, '$.shipping.method') FROM orders LIMIT 5; +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | id | customer_id | created_at | order_data | json_extract(orders.order_data, '$.shipping.method') | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 1 | 1 | 2025-11-21 13:08:22 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | standard | | | | | e}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 2 | 1 | 2025-11-21 18:42:58 | {"shipping": {"method": "overnight"}, "risk": {"flag": "high | overnight | | | | | "}, "gift": {"wrapped": false}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 3 | 1 | 2025-11-21 21:01:46 | {"shipping": {"method": "standard"}, "gift": {"wrapped": fal | standard | | | | | se}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 4 | 1 | 2025-11-24 13:17:27 | {"shipping": {"method": "standard"}, "gift": {"wrapped": tru | standard | | | | | e}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ | 5 | 1 | 2025-11-24 21:09:46 | {"shipping": {"method": "overnight"}, "gift": {"wrapped": fa | overnight | | | | | lse}} | | +----+-------------+---------------------+--------------------------------------------------------------+------------------------------------------------------+ sqlite> ``` Let's grab the other two. ```sql SELECT json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders; ``` ``` sqlite> select json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders LIMIT 5; +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ | json_extract(order_data, '$.shipping.method') | json_extract(order_data, '$.gift.wrapped') | json_extract(order_data, '$.risk.flag') | +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ | standard | 1 | | | overnight | 0 | high | | standard | 0 | | | standard | 1 | | | overnight | 0 | | +-----------------------------------------------+--------------------------------------------+-----------------------------------------+ sqlite> ``` Hmm, interesting, it automatically converted `false` to `0`, and `true` to `1` as you can see in the actual json data it was `true` and `false`. Nice, but could go the other way as well, if you were checking for true or false instead of 0 and 1. Both are same in SQLite, they are treated as truth and false values. We have all the things we need, now? Well, let's group by the `customer_id` and order by the `created_at` to get the info about each customer with its latest order. ```sql SELECT *, json_extract(order_data, '$.shipping.method'), json_extract(order_data, '$.gift.wrapped'), json_extract(order_data, '$.risk.flag') FROM orders GROUP BY customer_id ORDER by created_at LIMIT 5; ``` But there I see the problem, how can we group by customer_id and find the latest order for that customer! The order by will only work after the grouping has been done, so its not necessarily the per order for each customer. Hmm! We need to make a subquery to get the latest order for each customer it seems. ```sql SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag FROM orders WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ORDER BY orders.created_at DESC; ``` We just added this ```sql WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ``` This will filter only the latest created order for each customer and then we can grab the details from json in each row per customer after this isolates the row with the latest created at time. ``` sqlite> SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag FROM orders WHERE orders.created_at = ( SELECT MAX(created_at) FROM orders AS latest_order WHERE orders.customer_id = latest_order.customer_id ) ORDER BY orders.created_at DESC; +-------------+---------------------+-----------------+-----------+-----------+ | customer_id | created_at | shipping_method | gift_wrap | risk_flag | +-------------+---------------------+-----------------+-----------+-----------+ | 32 | 2025-12-17 21:17:39 | overnight | 0 | | | 15 | 2025-12-17 19:21:33 | express | 0 | medium | | 50 | 2025-12-17 14:47:54 | express | 1 | low | | 43 | 2025-12-17 14:23:46 | express | 1 | | | 27 | 2025-12-17 14:05:13 | standard | 1 | | | 3 | 2025-12-17 14:02:28 | standard | 1 | high | | 49 | 2025-12-17 13:28:49 | express | 1 | high | | 36 | 2025-12-17 11:11:29 | overnight | 1 | | | 31 | 2025-12-17 08:05:46 | express | 0 | | | 16 | 2025-12-17 07:32:36 | express | 0 | | | 38 | 2025-12-17 06:05:12 | standard | 1 | | | 44 | 2025-12-17 05:28:54 | standard | 1 | | | 9 | 2025-12-17 04:33:08 | express | 1 | | | 23 | 2025-12-17 03:01:49 | express | 0 | | | 21 | 2025-12-16 23:53:14 | overnight | 1 | | | 25 | 2025-12-16 20:49:58 | overnight | 1 | high | | 46 | 2025-12-16 19:38:37 | standard | 0 | | | 1 | 2025-12-16 19:23:53 | express | 0 | | | 28 | 2025-12-16 18:20:55 | standard | 0 | low | | 40 | 2025-12-16 17:54:05 | express | 0 | | | 13 | 2025-12-16 16:11:16 | standard | 1 | | | 24 | 2025-12-16 14:19:45 | overnight | 0 | | | 11 | 2025-12-16 11:20:31 | standard | 1 | medium | | 17 | 2025-12-16 08:19:36 | standard | 0 | | | 4 | 2025-12-16 04:38:51 | express | 0 | | | 34 | 2025-12-16 02:11:57 | express | 0 | | | 30 | 2025-12-15 15:32:04 | overnight | 0 | medium | | 48 | 2025-12-15 13:03:59 | standard | 1 | | | 41 | 2025-12-15 13:00:00 | standard | 0 | high | | 45 | 2025-12-15 11:37:57 | standard | 0 | | | 7 | 2025-12-14 23:39:47 | express | 0 | | | 35 | 2025-12-14 22:46:36 | express | 1 | high | | 47 | 2025-12-14 20:53:07 | standard | 0 | | | 22 | 2025-12-14 12:38:58 | standard | 0 | medium | | 12 | 2025-12-14 07:59:28 | standard | 1 | medium | | 18 | 2025-12-14 04:55:34 | overnight | 0 | low | | 20 | 2025-12-14 04:54:07 | overnight | 0 | | | 14 | 2025-12-13 07:44:19 | standard | 1 | | | 6 | 2025-12-13 07:03:12 | overnight | 1 | | | 10 | 2025-12-13 04:23:37 | standard | 0 | medium | | 19 | 2025-12-13 03:29:15 | standard | 0 | | | 8 | 2025-12-12 12:42:18 | overnight | 0 | | | 26 | 2025-12-11 17:35:46 | standard | 0 | low | | 37 | 2025-12-11 13:55:35 | overnight | 1 | | | 33 | 2025-12-09 12:30:54 | express | 1 | | | 2 | 2025-12-08 07:17:31 | standard | 0 | | | 42 | 2025-12-08 02:48:12 | overnight | 0 | medium | | 5 | 2025-12-06 17:53:53 | overnight | 1 | | | 39 | 2025-12-06 14:38:29 | overnight | 1 | | | 29 | 2025-12-03 05:10:32 | overnight | 1 | high | +-------------+---------------------+-----------------+-----------+-----------+ sqlite> ``` Now, that is it! I guess, a quick solution for getting this data sorted. However there is one more way without needing the subquery. ### ROW NUMBER - Window Function We can create partition for each customer with the `customer_id` and then sort by the `created_at` date in a latest first order (descending) and simply take the first row for extraction. This is kind of the same thing, but a kind of 'gives a good feel to me' solution. Elite Mindset! ```sql WITH ranked_orders AS ( SELECT orders.customer_id, orders.created_at, json_extract(orders.order_data, '$.shipping.method') AS shipping_method, json_extract(orders.order_data, '$.gift.wrapped') AS gift_wrap, json_extract(orders.order_data, '$.risk.flag') AS risk_flag, ROW_NUMBER() OVER (PARTITION BY orders.customer_id ORDER BY orders.created_at DESC) AS row_num FROM orders ) SELECT customer_id, created_at, shipping_method, gift_wrap, risk_flag FROM ranked_orders WHERE row_num = 1 ORDER BY created_at DESC; ``` We are just doing the same thing as explained. This thing `ROW_NUMBER() OVER (PARTITION BY orders.customer_id ORDER BY orders.created_at DESC) AS row_num` It will partition the table `orders` with `customer_id` and order by `created_at` latest first. And we assign each row as the `row_number` and grab the 1st row per customer to grab the latest order (filter with WHERE when calling the cte, or querying. However we created the CTE `ranked_orders` or you can call it `latest_orders`, to get the `row_num` filter to only the first `1` and for that we grab the already extracted json data as well the other columns. Simple! That is it! Day 9 was easy peasy! Onwards day 10! Catch you tomorrow! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-8 Advent of SQL - Day 8, Product Catalog Whopsies! This is day 8. Let's get straigh... HOOH! We need to clean up some SQL for running in SQLite. Just cleaning up Tue, 23 Dec 2025 00:00:00 UTC ## Advent of SQL - Day 8, Product Catalog Whopsies! This is day 8. Let's get straigh... HOOH! We need to clean up some SQL for running in SQLite. ```bash sed -i 's/TIMESTAMP[[:space:]]*//g' day8-inserts-sqlite.sql ``` Just cleaning up `TIMESTAMP` in `INSERT` before the date value. Here we go: The SQL to run in SQLite. ```sql DROP TABLE IF EXISTS price_changes; DROP TABLE IF EXISTS products; CREATE TABLE products ( product_id INT PRIMARY KEY, product_name TEXT ); CREATE TABLE price_changes ( id INT PRIMARY KEY, product_id INT, price NUMERIC(10,2), effective_timestamp ); INSERT INTO products (product_id, product_name) VALUES (1, 'Deluxe Sled'), (2, 'Holiday Trail Mix Trio'), (3, 'Premium Cinnamon Roasted Almonds'), (4, 'Deluxe Wrapping Paper'), (5, 'Deluxe Roasted Cashews'), (6, 'Festive Cookware Set'), (7, 'Deluxe Mug'), (8, 'Premium Sled'), (9, 'Essential Sled'), (10, 'Family Snow Boots'), (11, 'Family Dark Chocolate Almonds'), (12, 'Premium Festive Scarf'), (13, 'Essential Cookie Decorating Kit'), (14, 'Festive White Chocolate Popcorn'), (15, 'Cozy Puzzle'), (16, 'Holiday Cheddar Popcorn'), (17, 'Premium Board Game'), (18, 'Deluxe Pecan Praline Bites'), (19, 'Cozy Almond Brittle'), (20, 'Winter Sled'); INSERT INTO price_changes (id, product_id, price, effective_timestamp) VALUES (1, 1, 148.28, '2025-12-01 05:25:35'), (2, 1, 148.63, '2025-12-02 18:15:33'), (3, 1, 126.78, '2025-12-02 18:40:38'), (4, 1, 119.12, '2025-12-03 10:14:35'), (5, 1, 98.57, '2025-12-04 04:14:31'), (6, 1, 88.49, '2025-12-06 19:02:40'), (7, 1, 80.88, '2025-12-07 10:43:54'), (8, 1, 78.88, '2025-12-08 06:45:39'), (9, 1, 80.24, '2025-12-08 16:11:11'), (10, 1, 73.9, '2025-12-10 14:33:43'), (11, 1, 88.2, '2025-12-12 02:21:09'), (12, 1, 99.03, '2025-12-12 02:58:14'), (13, 1, 100.18, '2025-12-14 15:58:03'), (14, 1, 106.91, '2025-12-16 01:51:05'), (15, 1, 109.25, '2025-12-16 16:01:53'), (16, 2, 29.54, '2025-12-03 14:21:10'), (17, 2, 34.33, '2025-12-03 19:14:31'), (18, 2, 39.08, '2025-12-04 06:13:48'), (19, 2, 32.71, '2025-12-04 18:33:17'), (20, 2, 31.71, '2025-12-05 22:36:14'), (21, 2, 28.88, '2025-12-06 02:42:02'), (22, 2, 23.14, '2025-12-07 09:46:54'), (23, 2, 25.65, '2025-12-07 10:03:45'), (24, 2, 27.06, '2025-12-07 14:39:15'), (25, 2, 24.48, '2025-12-07 20:08:05'), (26, 2, 26.84, '2025-12-09 07:44:32'), (27, 2, 27.39, '2025-12-13 06:25:19'), (28, 2, 26.6, '2025-12-14 10:16:34'), (29, 2, 21.38, '2025-12-15 16:20:20'), (30, 2, 17.75, '2025-12-16 09:28:13'); ``` We can get started. ## Problem > Generate a report, using the products and `price_changes` tables for leadership that returns the `product_name`, `current_price`, `previous_price`, and the difference between the current and previous prices So what we need is - product_name - current_price (latest) - previous_price (just before the latest) - price_difference = current - previous Again we have to meddle with dates, maybe, maybe not! ### With CTEs and JOINs Let's start with the simplest approach. We need 2 prices, the latest(highest date timestamp) and the 2nd latest (the 2nd highest date timestamp). We can get the first pretty easily, but what about the second? Well, if we get the first, then the second should be easy to get right? Right? Because it will be just before it. Well not directly. Let's first grab the max timestamp. ```sql SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id; ``` ``` sqlite> SELECT product_id, MAX(effective_timestamp) AS max_ts FROM price_changes GROUP BY product_id; +------------+---------------------+ | product_id | max_ts | +------------+---------------------+ | 1 | 2025-12-16 16:01:53 | | 2 | 2025-12-16 09:28:13 | | 3 | 2025-12-15 03:20:11 | | 4 | 2025-12-16 01:33:41 | | 5 | 2025-12-12 10:11:48 | | 6 | 2025-12-15 11:31:40 | | 7 | 2025-12-16 03:00:51 | | 8 | 2025-12-15 22:33:48 | | 9 | 2025-12-15 20:05:34 | | 10 | 2025-12-15 20:53:45 | ... ... | 139 | 2025-12-16 04:46:33 | | 140 | 2025-12-16 21:19:30 | | 141 | 2025-12-15 09:50:36 | | 142 | 2025-12-16 18:39:51 | | 143 | 2025-12-15 07:27:06 | | 144 | 2025-12-16 16:25:16 | | 146 | 2025-12-13 07:07:19 | | 148 | 2025-12-16 09:30:11 | | 149 | 2025-12-13 16:40:21 | | 150 | 2025-12-13 08:24:43 | +------------+---------------------+ ``` We got all the timestamp's for each product. But we wanted the prices. Well we can't grab the price here, since we are grouping! We can use the timestamp as we are selecting the `MAX` aggregate function however among the many rows for single product how do we group the price? `MIN`, `MAX`, `AVG`, but that's not we want, we just want the price for that timestamp. Well, we need to join to the same table for that timestamp and grab the price. ```sql SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_ts FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id ) AS latest_price_change ON price_changes.product_id = latest_price_change.product_id AND price_changes.effective_timestamp = latest_price_change.latest_timestamp; ``` Here, we first specify what we want - `product_id` - `current_price` which is the price for the - `latest_timestamp` which is the latest time recorded for the product price. We are grouping this by `product_id` since there are price recorded and various timestamps. We need to get the latest timestamp. So, we do a nested query to join the latest timestamp and join the price with the same timestamp. This condition `price_changes.effective_timestamp = latest_price_change.latest_timestamp` helps us get the `price` for the `latest_timestamp`. We first get each timestamp for the product and then find its max, so that's the inner query from which we joined this table to. Self join with the different thing to find. This gives us 2 things. - Product id - Price for the latest timestamp We don't really want the timestamp in the final result, its just a criteria or a intermediate value to get the current and the previous prices for each product. ``` sqlite> sqlite> SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_ts FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS latest_timestamp FROM price_changes GROUP BY product_id ) AS latest_price_change ON price_changes.product_id = latest_price_change.product_id AND price_changes.effective_timestamp = latest_price_change.latest_timestamp; +------------+---------------+---------------------+ | product_id | current_price | latest_ts | +------------+---------------+---------------------+ | 1 | 109.25 | 2025-12-16 16:01:53 | | 2 | 17.75 | 2025-12-16 09:28:13 | | 3 | 143.65 | 2025-12-15 03:20:11 | | 4 | 98.51 | 2025-12-16 01:33:41 | | 5 | 124.04 | 2025-12-12 10:11:48 | | 6 | 84.14 | 2025-12-15 11:31:40 | | 7 | 123.09 | 2025-12-16 03:00:51 | | 8 | 221.06 | 2025-12-15 22:33:48 | | 9 | 255.88 | 2025-12-15 20:05:34 | | 10 | 57.99 | 2025-12-15 20:53:45 | ... ... | 139 | 16.41 | 2025-12-16 04:46:33 | | 140 | 173.05 | 2025-12-16 21:19:30 | | 141 | 69.97 | 2025-12-15 09:50:36 | | 142 | 35.05 | 2025-12-16 18:39:51 | | 143 | 153.94 | 2025-12-15 07:27:06 | | 144 | 118.21 | 2025-12-16 16:25:16 | | 146 | 54.73 | 2025-12-13 07:07:19 | | 148 | 107.81 | 2025-12-16 09:30:11 | | 149 | 72.6 | 2025-12-13 16:40:21 | | 150 | 138.66 | 2025-12-13 08:24:43 | +------------+---------------+---------------------+ sqlite> ``` Now, we need to the price before this max timestamp. How do we get it? We need to again join? Yes... We need to subquery the timestamp just before it. But how will we get the max timestamp for each product. Well that's what we wrote above. We can convert that to a CTE. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT * FROM latest; ``` We just got the same thing, however we can now use `latest` as a temporary table in the query. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` So, ok, this is getting long. Just we added this ```sql SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ``` This won't work as we need the latest table which we converted to CTE. So, to get the 2nd highest or latest timestamp for each product, we do this. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id; ``` We use: ```sql JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp ``` To get match the timestamp which will exclude the latest timestamp and that way we can again get the `MAX(price_changes.effective_timestamp) AS previous_timestamp` for the subset of the timestamp. This gives us all the previous timestamps i.e. the price just before the max timestamp for each product. ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ) SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id; +------------+----------------+---------------------+ | product_id | previous_price | previous_timestamp | +------------+----------------+---------------------+ | 1 | 106.91 | 2025-12-16 01:51:05 | | 2 | 21.38 | 2025-12-15 16:20:20 | | 3 | 159.65 | 2025-12-14 09:52:09 | | 4 | 105.6 | 2025-12-14 14:09:20 | | 5 | 129.23 | 2025-12-12 04:08:45 | | 6 | 88.97 | 2025-12-15 02:13:04 | | 7 | 127.14 | 2025-12-13 07:25:12 | | 8 | 241.99 | 2025-12-14 19:31:40 | | 9 | 259.56 | 2025-12-12 12:47:13 | | 10 | 64.88 | 2025-12-15 10:52:34 | ... ... | 140 | 157.02 | 2025-12-09 17:58:07 | | 141 | 73.88 | 2025-12-13 14:35:53 | | 142 | 30.25 | 2025-12-16 13:44:42 | | 143 | 143.04 | 2025-12-11 10:08:13 | | 144 | 114.22 | 2025-12-15 17:33:37 | | 146 | 65.71 | 2025-12-10 15:50:14 | | 148 | 101.09 | 2025-12-15 05:37:27 | | 149 | 86.31 | 2025-12-13 13:18:14 | | 150 | 123.61 | 2025-12-12 10:49:22 | +------------+----------------+---------------------+ sqlite> ``` Now, we need to join both of these on the same product id, to fetch both previous and current timestamp as well as the price. ```sql SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` Simple ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM previous JOIN latest ON previous.product_id = latest.product_id; +------------+----------------+---------------------+------------+---------------+---------------------+ | product_id | previous_price | previous_timestamp | product_id | current_price | latest_timestamp | +------------+----------------+---------------------+------------+---------------+---------------------+ | 1 | 106.91 | 2025-12-16 01:51:05 | 1 | 109.25 | 2025-12-16 16:01:53 | | 2 | 21.38 | 2025-12-15 16:20:20 | 2 | 17.75 | 2025-12-16 09:28:13 | | 3 | 159.65 | 2025-12-14 09:52:09 | 3 | 143.65 | 2025-12-15 03:20:11 | | 4 | 105.6 | 2025-12-14 14:09:20 | 4 | 98.51 | 2025-12-16 01:33:41 | | 5 | 129.23 | 2025-12-12 04:08:45 | 5 | 124.04 | 2025-12-12 10:11:48 | | 6 | 88.97 | 2025-12-15 02:13:04 | 6 | 84.14 | 2025-12-15 11:31:40 | | 7 | 127.14 | 2025-12-13 07:25:12 | 7 | 123.09 | 2025-12-16 03:00:51 | | 8 | 241.99 | 2025-12-14 19:31:40 | 8 | 221.06 | 2025-12-15 22:33:48 | | 9 | 259.56 | 2025-12-12 12:47:13 | 9 | 255.88 | 2025-12-15 20:05:34 | | 10 | 64.88 | 2025-12-15 10:52:34 | 10 | 57.99 | 2025-12-15 20:53:45 | ... ... | 139 | 15.26 | 2025-12-12 03:43:32 | 139 | 16.41 | 2025-12-16 04:46:33 | | 140 | 157.02 | 2025-12-09 17:58:07 | 140 | 173.05 | 2025-12-16 21:19:30 | | 141 | 73.88 | 2025-12-13 14:35:53 | 141 | 69.97 | 2025-12-15 09:50:36 | | 142 | 30.25 | 2025-12-16 13:44:42 | 142 | 35.05 | 2025-12-16 18:39:51 | | 143 | 143.04 | 2025-12-11 10:08:13 | 143 | 153.94 | 2025-12-15 07:27:06 | | 144 | 114.22 | 2025-12-15 17:33:37 | 144 | 118.21 | 2025-12-16 16:25:16 | | 146 | 65.71 | 2025-12-10 15:50:14 | 146 | 54.73 | 2025-12-13 07:07:19 | | 148 | 101.09 | 2025-12-15 05:37:27 | 148 | 107.81 | 2025-12-16 09:30:11 | | 149 | 86.31 | 2025-12-13 13:18:14 | 149 | 72.6 | 2025-12-13 16:40:21 | | 150 | 123.61 | 2025-12-12 10:49:22 | 150 | 138.66 | 2025-12-13 08:24:43 | +------------+----------------+---------------------+------------+---------------+---------------------+ sqlite> ``` Now, we are getting somewhere, we just need to find the difference right? Yes, but more JOINs We need the product_name from the `product` table. Almost forgot that table exists? ```sql SELECT * FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` We need both right? So, we need to fetch the product id and join for the latest and the previous table from the CTE, joining on the `product_id`. ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT * FROM products JOIN latest ON products.product_id = latest.product_id ...> LEFT JOIN previous ON products.product_id = previous.product_id; +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ | product_id | product_name | product_id | current_price | latest_timestamp | product_id | previous_price | previous_timestamp | +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ | 1 | Deluxe Sled | 1 | 109.25 | 2025-12-16 16:01:53 | 1 | 106.91 | 2025-12-16 01:51:05 | | 2 | Holiday Trail Mix Trio | 2 | 17.75 | 2025-12-16 09:28:13 | 2 | 21.38 | 2025-12-15 16:20:20 | | 3 | Premium Cinnamon Roasted Almonds | 3 | 143.65 | 2025-12-15 03:20:11 | 3 | 159.65 | 2025-12-14 09:52:09 | | 4 | Deluxe Wrapping Paper | 4 | 98.51 | 2025-12-16 01:33:41 | 4 | 105.6 | 2025-12-14 14:09:20 | | 5 | Deluxe Roasted Cashews | 5 | 124.04 | 2025-12-12 10:11:48 | 5 | 129.23 | 2025-12-12 04:08:45 | | 6 | Festive Cookware Set | 6 | 84.14 | 2025-12-15 11:31:40 | 6 | 88.97 | 2025-12-15 02:13:04 | | 7 | Deluxe Mug | 7 | 123.09 | 2025-12-16 03:00:51 | 7 | 127.14 | 2025-12-13 07:25:12 | | 8 | Premium Sled | 8 | 221.06 | 2025-12-15 22:33:48 | 8 | 241.99 | 2025-12-14 19:31:40 | | 9 | Essential Sled | 9 | 255.88 | 2025-12-15 20:05:34 | 9 | 259.56 | 2025-12-12 12:47:13 | | 10 | Family Snow Boots | 10 | 57.99 | 2025-12-15 20:53:45 | 10 | 64.88 | 2025-12-15 10:52:34 | ... ... | 140 | Classic Mug | 140 | 173.05 | 2025-12-16 21:19:30 | 140 | 157.02 | 2025-12-09 17:58:07 | | 141 | Family Fruit Assortment | 141 | 69.97 | 2025-12-15 09:50:36 | 141 | 73.88 | 2025-12-13 14:35:53 | | 142 | Classic Ornament | 142 | 35.05 | 2025-12-16 18:39:51 | 142 | 30.25 | 2025-12-16 13:44:42 | | 143 | Essential Ornament | 143 | 153.94 | 2025-12-15 07:27:06 | 143 | 143.04 | 2025-12-11 10:08:13 | | 144 | Premium Trail Mix Trio | 144 | 118.21 | 2025-12-16 16:25:16 | 144 | 114.22 | 2025-12-15 17:33:37 | | 146 | Premium Book Collection | 146 | 54.73 | 2025-12-13 07:07:19 | 146 | 65.71 | 2025-12-10 15:50:14 | | 148 | Cozy Trail Mix Trio | 148 | 107.81 | 2025-12-16 09:30:11 | 148 | 101.09 | 2025-12-15 05:37:27 | | 149 | Family Cheddar Popcorn | 149 | 72.6 | 2025-12-13 16:40:21 | 149 | 86.31 | 2025-12-13 13:18:14 | | 150 | Holiday Headphones | 150 | 138.66 | 2025-12-13 08:24:43 | 150 | 123.61 | 2025-12-12 10:49:22 | +------------+-----------------------------------+------------+---------------+---------------------+------------+----------------+---------------------+ sqlite> ``` Now, we just want - product_name - previous_price - current_price - difference of current_price and previous price So, ```sql WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT products.product_name, latest.current_price, previous.previous_price, (latest.current_price - previous.previous_price) as price_difference FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; ``` ``` sqlite> WITH latest AS ( SELECT price_changes.product_id, price_changes.price AS current_price, price_changes.effective_timestamp AS latest_timestamp FROM price_changes JOIN ( SELECT product_id, MAX(effective_timestamp) AS max_timestamp FROM price_changes GROUP BY product_id ) latest_price_changes ON price_changes.product_id = latest_price_changes.product_id AND price_changes.effective_timestamp = latest_price_changes.max_timestamp ), previous AS ( SELECT price_changes.product_id, price_changes.price AS previous_price, MAX(price_changes.effective_timestamp) AS previous_timestamp FROM price_changes JOIN latest ON price_changes.product_id = latest.product_id WHERE price_changes.effective_timestamp < latest.latest_timestamp GROUP BY price_changes.product_id ) SELECT products.product_name, latest.current_price, previous.previous_price, (latest.current_price - previous.previous_price) as price_difference FROM products JOIN latest ON products.product_id = latest.product_id LEFT JOIN previous ON products.product_id = previous.product_id; +-----------------------------------+---------------+----------------+-------------------+ | product_name | current_price | previous_price | price_difference | +-----------------------------------+---------------+----------------+-------------------+ | Deluxe Sled | 109.25 | 106.91 | 2.34 | | Holiday Trail Mix Trio | 17.75 | 21.38 | -3.63 | | Premium Cinnamon Roasted Almonds | 143.65 | 159.65 | -16.0 | | Deluxe Wrapping Paper | 98.51 | 105.6 | -7.08999999999999 | | Deluxe Roasted Cashews | 124.04 | 129.23 | -5.18999999999998 | | Festive Cookware Set | 84.14 | 88.97 | -4.83 | | Deluxe Mug | 123.09 | 127.14 | -4.05 | | Premium Sled | 221.06 | 241.99 | -20.93 | | Essential Sled | 255.88 | 259.56 | -3.68000000000001 | ... ... | Classic Ornament | 35.05 | 30.25 | 4.8 | | Essential Ornament | 153.94 | 143.04 | 10.9 | | Premium Trail Mix Trio | 118.21 | 114.22 | 3.98999999999999 | | Premium Book Collection | 54.73 | 65.71 | -10.98 | | Cozy Trail Mix Trio | 107.81 | 101.09 | 6.72 | | Family Cheddar Popcorn | 72.6 | 86.31 | -13.71 | | Holiday Headphones | 138.66 | 123.61 | 15.05 | +-----------------------------------+---------------+----------------+-------------------+ sqlite> ``` Oh ok! That is it! CTEs and some nasty number of JOINs. Can we do it some other ways? Of course we can, and we will ### ROW NUMBER - Window Function We can even use some window functions because we are doing things per product so we can leverage [ROW_NUMBER](https://sqlite.org/windowfunctions.html#:~:text=in%20window%20functions%3A-,row_number(),-The%20number%20of) in our case. > ROW_NUMBER: The number of the row within the current partition. Rows are numbered starting from 1 in the order defined by the ORDER BY clause in the window definition, or in arbitrary order otherwise. So, we can partition by `product_id` in the `price_changes` table and order by the timestamp (latest first i.e. descending) and grab the first two prices. ```sql WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT * FROM ranked_prices; ``` Here, we just partitioned the price table based on the latest to oldest timestamp (descending) and added the other column `row_number` that will be further used to get the `previous` and the `current` price and time. ``` sqlite> WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT * FROM ranked_prices; +------+------------+--------+---------------------+------------+ | id | product_id | price | effective_timestamp | row_number | +------+------------+--------+---------------------+------------+ | 15 | 1 | 109.25 | 2025-12-16 16:01:53 | 1 | | 14 | 1 | 106.91 | 2025-12-16 01:51:05 | 2 | | 13 | 1 | 100.18 | 2025-12-14 15:58:03 | 3 | | 12 | 1 | 99.03 | 2025-12-12 02:58:14 | 4 | | 11 | 1 | 88.2 | 2025-12-12 02:21:09 | 5 | | 10 | 1 | 73.9 | 2025-12-10 14:33:43 | 6 | | 9 | 1 | 80.24 | 2025-12-08 16:11:11 | 7 | | 8 | 1 | 78.88 | 2025-12-08 06:45:39 | 8 | | 7 | 1 | 80.88 | 2025-12-07 10:43:54 | 9 | | 6 | 1 | 88.49 | 2025-12-06 19:02:40 | 10 | | 5 | 1 | 98.57 | 2025-12-04 04:14:31 | 11 | | 4 | 1 | 119.12 | 2025-12-03 10:14:35 | 12 | | 3 | 1 | 126.78 | 2025-12-02 18:40:38 | 13 | | 2 | 1 | 148.63 | 2025-12-02 18:15:33 | 14 | | 1 | 1 | 148.28 | 2025-12-01 05:25:35 | 15 | ... ... | 1260 | 149 | 167.1 | 2025-12-01 06:36:43 | 15 | | 1288 | 150 | 138.66 | 2025-12-13 08:24:43 | 1 | | 1287 | 150 | 123.61 | 2025-12-12 10:49:22 | 2 | | 1286 | 150 | 141.8 | 2025-12-10 07:06:06 | 3 | | 1285 | 150 | 122.16 | 2025-12-09 20:01:54 | 4 | | 1284 | 150 | 122.06 | 2025-12-06 22:27:41 | 5 | | 1283 | 150 | 128.6 | 2025-12-06 13:03:18 | 6 | | 1282 | 150 | 154.72 | 2025-12-05 08:15:08 | 7 | | 1281 | 150 | 170.3 | 2025-12-04 15:25:50 | 8 | | 1280 | 150 | 156.51 | 2025-12-03 19:11:12 | 9 | | 1279 | 150 | 161.93 | 2025-12-02 02:47:10 | 10 | | 1278 | 150 | 174.36 | 2025-12-02 01:39:14 | 11 | | 1277 | 150 | 180.17 | 2025-12-01 20:36:02 | 12 | | 1276 | 150 | 164.35 | 2025-12-01 07:09:29 | 13 | | 1275 | 150 | 141 | 2025-12-01 01:29:46 | 14 | +------+------------+--------+---------------------+------------+ ``` ```sql WITH ranked_prices AS ( SELECT price_changes.*, ROW_NUMBER() OVER ( PARTITION BY price_changes.product_id ORDER BY price_changes.effective_timestamp DESC ) AS row_number FROM price_changes ) SELECT products.product_name, current_price.price AS current_price, previous_price.price AS previous_price, current_price.price - previous_price.price AS price_difference FROM products LEFT JOIN ranked_prices AS current_price ON products.product_id = current_price.product_id AND current_price.row_number = 1 LEFT JOIN ranked_prices AS previous_price ON products.product_id = previous_price.product_id AND previous_price.row_number = 2; ``` This is pretty simple, we just get - the `current_price` as `row_number = 1` from the `ranked_prices` CTE - the `previous_price` as `row_number = 2` from the `ranked_prices` CTE We got all we needed as: - `products.product_name` - `current_price.price AS current_price` - `previous_price.price AS previous_price` - `current_price.price - previous_price.price AS price_difference` So, this is how `ROW_NUMBER` can be used here. ### With LEAD LAG - Window Functions We can also use [LEAD](https://sqlite.org/windowfunctions.html#:~:text=does%20not%20exist.-,lead(expr),-lead(expr%2C%20offset) and [LAG](https://sqlite.org/windowfunctions.html#:~:text=a%20part%20of.-,lag(expr),-lag(expr%2C%20offset) window functions here. We can specifically use `LAG` as it was a challenge to get the second latest timestamped price. We will partition the `price_changes` table on the `product_id` and order it by the `effective_timestamp`. This will give us the row before the timestamp of the current one, so the first row for each product can be empty (if sorted in ascending order of timestamp since there is no before row for the first row) ```sql WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT * FROM lagged_prices; ``` Here, we simply selected the data that we need, `product_id`, `price` as the `current_price` since we can order by latest timestamp. ``` sqlite> WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT * FROM lagged_prices; +------------+---------------+----------------+---------------------+ | product_id | current_price | previous_price | effective_timestamp | +------------+---------------+----------------+---------------------+ | 1 | 148.28 | | 2025-12-01 05:25:35 | | 1 | 148.63 | 148.28 | 2025-12-02 18:15:33 | | 1 | 126.78 | 148.63 | 2025-12-02 18:40:38 | | 1 | 119.12 | 126.78 | 2025-12-03 10:14:35 | | 1 | 98.57 | 119.12 | 2025-12-04 04:14:31 | | 1 | 88.49 | 98.57 | 2025-12-06 19:02:40 | | 1 | 80.88 | 88.49 | 2025-12-07 10:43:54 | | 1 | 78.88 | 80.88 | 2025-12-08 06:45:39 | | 1 | 80.24 | 78.88 | 2025-12-08 16:11:11 | | 1 | 73.9 | 80.24 | 2025-12-10 14:33:43 | | 1 | 88.2 | 73.9 | 2025-12-12 02:21:09 | | 1 | 99.03 | 88.2 | 2025-12-12 02:58:14 | | 1 | 100.18 | 99.03 | 2025-12-14 15:58:03 | | 1 | 106.91 | 100.18 | 2025-12-16 01:51:05 | | 1 | 109.25 | 106.91 | 2025-12-16 16:01:53 | | 2 | 29.54 | | 2025-12-03 14:21:10 | | 2 | 34.33 | 29.54 | 2025-12-03 19:14:31 | | 2 | 39.08 | 34.33 | 2025-12-04 06:13:48 | | 2 | 32.71 | 39.08 | 2025-12-04 18:33:17 | | 2 | 31.71 | 32.71 | 2025-12-05 22:36:14 | | 2 | 28.88 | 31.71 | 2025-12-06 02:42:02 | ... ... | 149 | 101.03 | 98.4 | 2025-12-10 00:37:46 | | 149 | 95.68 | 101.03 | 2025-12-13 01:10:53 | | 149 | 86.31 | 95.68 | 2025-12-13 13:18:14 | | 149 | 72.6 | 86.31 | 2025-12-13 16:40:21 | | 150 | 141 | | 2025-12-01 01:29:46 | | 150 | 164.35 | 141 | 2025-12-01 07:09:29 | | 150 | 180.17 | 164.35 | 2025-12-01 20:36:02 | | 150 | 174.36 | 180.17 | 2025-12-02 01:39:14 | | 150 | 161.93 | 174.36 | 2025-12-02 02:47:10 | | 150 | 156.51 | 161.93 | 2025-12-03 19:11:12 | | 150 | 170.3 | 156.51 | 2025-12-04 15:25:50 | | 150 | 154.72 | 170.3 | 2025-12-05 08:15:08 | | 150 | 128.6 | 154.72 | 2025-12-06 13:03:18 | | 150 | 122.06 | 128.6 | 2025-12-06 22:27:41 | | 150 | 122.16 | 122.06 | 2025-12-09 20:01:54 | | 150 | 141.8 | 122.16 | 2025-12-10 07:06:06 | | 150 | 123.61 | 141.8 | 2025-12-12 10:49:22 | | 150 | 138.66 | 123.61 | 2025-12-13 08:24:43 | +------------+---------------+----------------+---------------------+ ``` Then we can join. The conditions are important to filter. - we get the lagged time stamp where we want the max of the timestamp since the latest timestamp will have the lagging timestamp for it from the CTE of lagged_prices and current_price as well. ```sql SELECT products.product_name, lagged_prices.current_price, lagged_prices.previous_price, lagged_prices.current_price - lagged_prices.previous_price AS price_difference FROM products JOIN lagged_prices ON products.product_id = lagged_prices.product_id WHERE lagged_prices.effective_timestamp = ( SELECT MAX(effective_timestamp) FROM price_changes WHERE product_id = products.product_id ); ``` ```sql WITH lagged_prices AS ( SELECT product_id, price AS current_price, LAG(price) OVER ( PARTITION BY product_id ORDER BY effective_timestamp ) AS previous_price, effective_timestamp FROM price_changes ) SELECT products.product_name, lagged_prices.current_price, lagged_prices.previous_price, lagged_prices.current_price - lagged_prices.previous_price AS price_difference FROM products JOIN lagged_prices ON products.product_id = lagged_prices.product_id WHERE lagged_prices.effective_timestamp = ( SELECT MAX(effective_timestamp) FROM price_changes WHERE product_id = products.product_id ); ``` This yields the same result. This is all big, any smaller queries? Not quite small. ### LIMIT OFFSET and CTE Well here's a little short, but quite dirty. ```sql SELECT *, current_price - previous_price AS price_difference FROM ( SELECT products.product_name, (SELECT price FROM price_changes WHERE price_changes.product_id = products.product_id ORDER BY effective_timestamp DESC LIMIT 1) AS current_price, (SELECT price FROM price_changes WHERE price_changes.product_id = products.product_id ORDER BY effective_timestamp DESC LIMIT 1 OFFSET 1) AS previous_price FROM products ); ``` This basically grabs the price from the latest timestamp and the 2nd timestamp with offset and wraps it in a CTE to compute the difference. Pretty slick if you ask me. But hey! I am done for this! It was a great problem. Getting data behind certain data is quite relatable and challenging enough to explore window functions and what not. That's it from day 8. See you tomorrow for day 9! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-7 Advent of SQL, Day 7 - Polar Express There were a few things, I had to dig up for converting the JSON in the statements into strings for SQLite, we can't really Mon, 22 Dec 2025 00:00:00 UTC ## Advent of SQL, Day 7 - Polar Express There were a few things, I had to dig up for converting the JSON `ARRAY[]` in the statements into strings for SQLite, we can't really use list of strings in SQLite. Here's the command to convert that array of strings into string. ``` sed "s/ARRAY\['/'\[\"/g; s/','/\",\"/g; s/']/\"]'/g" day7-inserts.sql > day7-inserts-sqlite.sql ``` OK, once that's done, this can be safely run into a sqlite database. ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day7-inserts-sqlite.sql sqlite> .schema CREATE TABLE passengers ( passenger_id INT PRIMARY KEY, passenger_name TEXT, favorite_mixins TEXT[], car_id INT ); CREATE TABLE cocoa_cars ( car_id INT PRIMARY KEY, available_mixins TEXT[], total_stock INT ); sqlite> sqlite> .mode table sqlite> select * from passengers limit 20; +--------------+----------------+--------------------------------------------------------------+--------+ | passenger_id | passenger_name | favorite_mixins | car_id | +--------------+----------------+--------------------------------------------------------------+--------+ | 1 | Ava Johnson | ["vanilla foam"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 5 | Layla Brooks | ["crispy rice","dark chocolate","caramel drizzle","cinnamon" | 3 | | | | ] | | +--------------+----------------+--------------------------------------------------------------+--------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 7 | Sofia Kim | ["cinnamon"] | 9 | +--------------+----------------+--------------------------------------------------------------+--------+ | 8 | Jonah Wolfe | ["cinnamon","dark chocolate"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | +--------------+----------------+--------------------------------------------------------------+--------+ | 10 | Diego Ramos | ["shaved chocolate"] | 1 | +--------------+----------------+--------------------------------------------------------------+--------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 12 | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | 8 | +--------------+----------------+--------------------------------------------------------------+--------+ | 13 | Mila Novak | ["crispy rice","cinnamon"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 14 | Lucas Ford | ["vanilla foam","white chocolate","cinnamon"] | 4 | +--------------+----------------+--------------------------------------------------------------+--------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 16 | Omar Qureshi | ["marshmallow"] | 3 | +--------------+----------------+--------------------------------------------------------------+--------+ | 17 | Keiko Ito | ["vanilla foam","marshmallow"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 18 | Tariq Hassan | ["dark chocolate","crispy rice","white chocolate","peppermin | 2 | | | | t"] | | +--------------+----------------+--------------------------------------------------------------+--------+ | 19 | Mira Zhao | ["caramel drizzle","marshmallow","cinnamon"] | 7 | +--------------+----------------+--------------------------------------------------------------+--------+ | 20 | Bianca Pereira | ["dark chocolate","peppermint"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ sqlite> sqlite> select * from cocoa_cars; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` Here's your full SQL file: ```sql DROP TABLE IF EXISTS passengers; DROP TABLE IF EXISTS cocoa_cars; CREATE TABLE passengers ( passenger_id INT PRIMARY KEY, passenger_name TEXT, favorite_mixins TEXT[], car_id INT ); CREATE TABLE cocoa_cars ( car_id INT PRIMARY KEY, available_mixins TEXT[], total_stock INT ); INSERT INTO passengers (passenger_id, passenger_name, favorite_mixins, car_id) VALUES (1, 'Ava Johnson', '["vanilla foam"]', 2), (2, 'Mateo Cruz', '["caramel drizzle","shaved chocolate","white chocolate"]', 2), (3, 'Nia Grant', '["shaved chocolate"]', 5), (4, 'Hiro Tanaka', '["shaved chocolate"]', 2), (5, 'Layla Brooks', '["crispy rice","dark chocolate","caramel drizzle","cinnamon"]', 3), (6, 'Ravi Patel', '["caramel drizzle","shaved chocolate","white chocolate"]', 5), (7, 'Sofia Kim', '["cinnamon"]', 9), (8, 'Jonah Wolfe', '["cinnamon","dark chocolate"]', 7), (9, 'Elena Morales', '["white chocolate","shaved chocolate","caramel drizzle"]', 6), (10, 'Diego Ramos', '["shaved chocolate"]', 1), (11, 'Zara Sheikh', '["vanilla foam","crispy rice","peppermint"]', 4), (12, 'Caleb Osei', '["shaved chocolate","dark chocolate","white chocolate"]', 8), (13, 'Mila Novak', '["crispy rice","cinnamon"]', 4), (14, 'Lucas Ford', '["vanilla foam","white chocolate","cinnamon"]', 4), (15, 'Yara Haddad', '["white chocolate","dark chocolate"]', 2), (16, 'Omar Qureshi', '["marshmallow"]', 3), (17, 'Keiko Ito', '["vanilla foam","marshmallow"]', 7), (18, 'Tariq Hassan', '["dark chocolate","crispy rice","white chocolate","peppermint"]', 2), (19, 'Mira Zhao', '["caramel drizzle","marshmallow","cinnamon"]', 7), (20, 'Bianca Pereira', '["dark chocolate","peppermint"]', 5), (21, 'Eva Schmidt', '["white chocolate","marshmallow"]', 5), (22, 'Rafael Silva', '["cinnamon","caramel drizzle"]', 3), (23, 'Nolan Murphy', '["caramel drizzle"]', 8), (24, 'Sara Johansson', '["marshmallow"]', 6), (25, 'Ingrid Nilsen', '["shaved chocolate","peppermint","marshmallow"]', 2), (26, 'Arjun Kapoor', '["dark chocolate"]', 2), (27, 'Nova Adams', '["cinnamon","dark chocolate"]', 9), (28, 'Felix Schneider', '["crispy rice","vanilla foam"]', 4), (29, 'Tim Cook', '["crispy rice","vanilla foam"]', 6), (30, 'Sophia Rossi', '["white chocolate","dark chocolate","peppermint","marshmallow"]', 4), (31, 'Liam OConnor', '["caramel drizzle"]', 1), (32, 'Olivia Dubois', '["shaved chocolate","peppermint"]', 2), (33, 'Emma Svensson', '["shaved chocolate"]', 2), (34, 'Noah Fischer', '["caramel drizzle"]', 2), (35, 'William Becker', '["crispy rice","dark chocolate"]', 4), (36, 'Isabella Laurent', '["dark chocolate","shaved chocolate"]', 8), (37, 'James Kim', '["shaved chocolate","marshmallow","peppermint","caramel drizzle"]', 7), (38, 'Mia Chen', '["vanilla foam","white chocolate","shaved chocolate"]', 1), (39, 'Benjamin Patel', '["peppermint"]', 7), (40, 'Charlotte Singh', '["marshmallow","shaved chocolate","white chocolate"]', 4), (41, 'Daniel Murphy', '["cinnamon","vanilla foam","marshmallow","white chocolate"]', 8), (42, 'Zoe Wilson', '["marshmallow","dark chocolate"]', 9), (43, 'Robert Smith', '["peppermint"]', 9), (44, 'Emily Johnson', '["marshmallow"]', 4), (45, 'David Brown', '["vanilla foam","dark chocolate"]', 8), (46, 'Sarah Davis', '["vanilla foam","peppermint"]', 3), (47, 'James Simon', '["peppermint","vanilla foam","cinnamon","shaved chocolate"]', 5), (48, 'Linda Lee', '["white chocolate","dark chocolate","marshmallow","vanilla foam"]', 5), (49, 'Carlos Mendez', '["peppermint","white chocolate"]', 6), (50, 'Fatima Noor', '["peppermint"]', 8), (51, 'Youssef El-Sayed', '["peppermint","marshmallow"]', 3), (52, 'Ian Landsman', '["marshmallow"]', 4), (53, 'Nolan Young', '["marshmallow","shaved chocolate","crispy rice","vanilla foam"]', 1), (54, 'Ava Martinez', '["vanilla foam"]', 9), (55, 'William Chen', '["crispy rice","shaved chocolate","caramel drizzle"]', 6), (56, 'Isabella Rodriguez', '["crispy rice","vanilla foam"]', 3), (57, 'Zachary Collins', '["dark chocolate","caramel drizzle","vanilla foam"]', 2), (58, 'Audrey Edwards', '["dark chocolate"]', 2), (59, 'Jason Stewart', '["white chocolate"]', 4), (60, 'Lucy Morris', '["cinnamon","caramel drizzle","peppermint"]', 4), (61, 'Cameron Rogers', '["crispy rice","cinnamon","shaved chocolate"]', 9), (62, 'Aria Blackwood', '["white chocolate","peppermint","caramel drizzle"]', 9), (63, 'Felix Whitmore', '["marshmallow","cinnamon","dark chocolate"]', 2), (64, 'Luna Hartley', '["white chocolate"]', 3), (65, 'Jasper Thorne', '["crispy rice","shaved chocolate","caramel drizzle"]', 6), (66, 'Nora Calloway', '["crispy rice","dark chocolate"]', 5), (67, 'Silas Merrick', '["marshmallow"]', 7), (68, 'Iris Pembroke', '["peppermint","white chocolate","cinnamon"]', 3), (69, 'Milo Ashford', '["cinnamon","dark chocolate","crispy rice"]', 7), (70, 'Clara Westbrook', '["marshmallow"]', 2), (71, 'Owen Fairchild', '["white chocolate","peppermint"]', 6), (72, 'Ruby Hawthorne', '["vanilla foam","cinnamon"]', 1), (73, 'Finn Lockhart', '["caramel drizzle","peppermint","cinnamon"]', 4), (74, 'Violet Sterling', '["marshmallow","caramel drizzle"]', 9), (75, 'August Blackwell', '["cinnamon","shaved chocolate","vanilla foam","marshmallow"]', 3), (76, 'Hazel Kincaid', '["peppermint","cinnamon","caramel drizzle","dark chocolate"]', 7), (77, 'Leo Greyson', '["crispy rice","cinnamon"]', 2), (78, 'Stella Beaumont', '["peppermint","dark chocolate","marshmallow","vanilla foam"]', 8), (79, 'Miles Brennan', '["crispy rice","shaved chocolate","marshmallow"]', 1), (80, 'Ivy Winslow', '["vanilla foam","caramel drizzle"]', 5), (81, 'Jack Carmichael', '["crispy rice"]', 6), (82, 'Scarlett Dalton', '["white chocolate","caramel drizzle","peppermint","vanilla foam"]', 5), (83, 'Oliver Ashby', '["crispy rice","peppermint"]', 2), (84, 'Aurora Whitfield', '["caramel drizzle","white chocolate","shaved chocolate","crispy rice"]', 9), (85, 'Noah Hastings', '["vanilla foam"]', 4), (86, 'Eliza Radcliffe', '["peppermint","vanilla foam","white chocolate"]', 9), (87, 'Liam Donovan', '["caramel drizzle","vanilla foam"]', 2), (88, 'Penelope Sinclair', '["caramel drizzle","marshmallow","white chocolate"]', 5), (89, 'Ethan Marlowe', '["vanilla foam","caramel drizzle","shaved chocolate"]', 5), (90, 'Charlotte Waverly', '["shaved chocolate","vanilla foam"]', 7), (91, 'Lucas Prescott', '["crispy rice","vanilla foam"]', 9), (92, 'Amelia Rosewood', '["crispy rice"]', 5), (93, 'Henry Treadwell', '["vanilla foam","caramel drizzle"]', 8), (94, 'Sophie Langford', '["dark chocolate","shaved chocolate","crispy rice","white chocolate"]', 3), (95, 'Benjamin Fairweather', '["crispy rice"]', 9), (96, 'Grace Aldridge', '["marshmallow","shaved chocolate","caramel drizzle"]', 5), (97, 'Samuel Kingsley', '["shaved chocolate","cinnamon"]', 1), (98, 'Eleanor Morrison', '["shaved chocolate"]', 8), (99, 'Daniel Lockwood', '["dark chocolate"]', 7), (100, 'Lucy Harrington', '["vanilla foam","dark chocolate"]', 7), (101, 'Matthew Sutherland', '["cinnamon","peppermint"]', 2), (102, 'Emma Gilmore', '["shaved chocolate","cinnamon","vanilla foam","caramel drizzle"]', 9), (103, 'Alexander Stratton', '["peppermint","shaved chocolate","vanilla foam","white chocolate"]', 8), (104, 'Abigail Worthington', '["dark chocolate","caramel drizzle"]', 8), (105, 'William Beauchamp', '["white chocolate","dark chocolate","marshmallow","caramel drizzle"]', 8), (106, 'Hannah Livingston', '["crispy rice","shaved chocolate","vanilla foam","caramel drizzle"]', 5), (107, 'James Garrison', '["shaved chocolate","crispy rice","white chocolate","peppermint"]', 5), (108, 'Sophia Brookshire', '["crispy rice","caramel drizzle"]', 6), (109, 'Theodore Hadley', '["cinnamon"]', 3), (110, 'Olivia Carrington', '["vanilla foam","cinnamon"]', 4), (111, 'Sebastian Ashworth', '["vanilla foam"]', 7), (112, 'Chloe Blackstone', '["white chocolate","dark chocolate","shaved chocolate"]', 1), (113, 'Nicholas Montague', '["vanilla foam","white chocolate"]', 1), (114, 'Madeline Ramsey', '["dark chocolate","peppermint","cinnamon","vanilla foam"]', 7), (115, 'Gabriel Winthrop', '["white chocolate","shaved chocolate","dark chocolate","marshmallow"]', 5), (116, 'Alice Merriweather', '["dark chocolate","peppermint","shaved chocolate","cinnamon"]', 7), (117, 'Isaac Kendrick', '["dark chocolate","cinnamon"]', 9), (118, 'Lillian Holbrook', '["vanilla foam"]', 1), (119, 'Caleb Bellamy', '["vanilla foam"]', 3), (120, 'Rose Drummond', '["cinnamon","peppermint","white chocolate","shaved chocolate"]', 6), (121, 'Elijah Wakefield', '["dark chocolate","caramel drizzle"]', 6), (122, 'Margaret Fairbanks', '["crispy rice","vanilla foam","cinnamon","peppermint"]', 8), (123, 'Julian Blackburn', '["white chocolate"]', 1), (124, 'Eva Templeton', '["shaved chocolate","marshmallow","vanilla foam"]', 1), (125, 'Nathan Whitley', '["shaved chocolate"]', 4), (126, 'Anna Westfield', '["cinnamon"]', 4), (127, 'Aaron Ashcroft', '["dark chocolate","marshmallow"]', 4), (128, 'Julia Pendleton', '["crispy rice","caramel drizzle","marshmallow","white chocolate"]', 2), (129, 'Connor Redmond', '["crispy rice","marshmallow"]', 1), (130, 'Grace Thornhill', '["vanilla foam","white chocolate","caramel drizzle"]', 4), (131, 'Zachary Stafford', '["shaved chocolate"]', 2), (132, 'Caroline Bannister', '["marshmallow","peppermint","cinnamon"]', 9), (133, 'Dylan Blakely', '["caramel drizzle","marshmallow","crispy rice","white chocolate"]', 6), (134, 'Katherine Underwood', '["vanilla foam"]', 8), (135, 'Tyler Braddock', '["vanilla foam"]', 6), (136, 'Victoria Harwood', '["cinnamon","vanilla foam","marshmallow","caramel drizzle"]', 9), (137, 'Ryan Beckett', '["white chocolate","dark chocolate","shaved chocolate"]', 7), (138, 'Elizabeth Chesterfield', '["caramel drizzle","shaved chocolate","vanilla foam"]', 2), (139, 'Jordan Waverly', '["dark chocolate","shaved chocolate","vanilla foam"]', 8), (140, 'Sarah Remington', '["caramel drizzle","peppermint","shaved chocolate","cinnamon"]', 3), (141, 'Brandon Locklear', '["shaved chocolate","caramel drizzle","vanilla foam","cinnamon"]', 6), (142, 'Rachel Wyndham', '["crispy rice","peppermint","white chocolate"]', 4), (143, 'Logan Sherwood', '["shaved chocolate"]', 7), (144, 'Amanda Fitzroy', '["white chocolate","shaved chocolate","caramel drizzle","dark chocolate"]', 8), (145, 'Jackson Thorpe', '["peppermint","marshmallow","cinnamon","dark chocolate"]', 7), (146, 'Rebecca Ashcombe', '["crispy rice","caramel drizzle"]', 8), (147, 'Cameron Gladstone', '["vanilla foam","caramel drizzle","cinnamon"]', 8), (148, 'Jessica Langston', '["crispy rice","white chocolate","marshmallow"]', 2), (149, 'Mason Fairmont', '["caramel drizzle","marshmallow"]', 9), (150, 'Emily Claridge', '["shaved chocolate","white chocolate"]', 8), (151, 'Hunter Bellingham', '["white chocolate","crispy rice","peppermint"]', 4), (152, 'Laura Thornbury', '["shaved chocolate","caramel drizzle","marshmallow"]', 5), (153, 'Wyatt Alderton', '["white chocolate"]', 3), (154, 'Claire Berkshire', '["peppermint","white chocolate","crispy rice"]', 5), (155, 'Cole Ashland', '["dark chocolate","marshmallow"]', 1), (156, 'Diana Brightwell', '["dark chocolate","white chocolate","shaved chocolate"]', 6), (157, 'Aiden Stanfield', '["peppermint","crispy rice"]', 8), (158, 'Natalie Warwick', '["marshmallow"]', 7), (159, 'Parker Blackmore', '["marshmallow","peppermint","white chocolate","vanilla foam"]', 5), (160, 'Morgan Steadman', '["shaved chocolate"]', 2), (161, 'Blake Dunwood', '["shaved chocolate","vanilla foam","white chocolate","dark chocolate"]', 5), (162, 'Taylor Woodridge', '["crispy rice","peppermint","white chocolate","caramel drizzle"]', 2), (163, 'Chase Ashbury', '["shaved chocolate","crispy rice"]', 2), (164, 'Madison Clearwater', '["shaved chocolate","cinnamon"]', 9), (165, 'Carter Brookfield', '["cinnamon"]', 1), (166, 'Ashley Fairhaven', '["dark chocolate","white chocolate","cinnamon","peppermint"]', 4), (167, 'Griffin Hartwell', '["crispy rice","dark chocolate","peppermint"]', 4), (168, 'Megan Redfield', '["shaved chocolate","vanilla foam","peppermint"]', 9), (169, 'Grayson Westmore', '["cinnamon","crispy rice"]', 3), (170, 'Nicole Ashridge', '["peppermint"]', 3), (171, 'Sawyer Hollingsworth', '["crispy rice","dark chocolate","peppermint"]', 8), (172, 'Alexis Thorndale', '["vanilla foam","crispy rice","dark chocolate"]', 9), (173, 'Declan Summerfield', '["dark chocolate","marshmallow","crispy rice","peppermint"]', 7), (174, 'Samantha Brightwood', '["crispy rice","peppermint","dark chocolate"]', 4), (175, 'Tristan Ashbrook', '["crispy rice"]', 1), (176, 'Melissa Ravenscroft', '["dark chocolate","white chocolate"]', 5), (177, 'Colton Hawthorne', '["vanilla foam","dark chocolate"]', 2), (178, 'Lauren Silverton', '["caramel drizzle","vanilla foam","cinnamon","dark chocolate"]', 2), (179, 'Landon Whitworth', '["caramel drizzle","vanilla foam"]', 8), (180, 'Kayla Mansfield', '["vanilla foam","peppermint","shaved chocolate"]', 2); INSERT INTO cocoa_cars (car_id, available_mixins, total_stock) VALUES (5, '["white chocolate","shaved chocolate"]', 412), (2, '["cinnamon","marshmallow","caramel drizzle"]', 359), (9, '["crispy rice","peppermint","caramel drizzle","shaved chocolate"]', 354), (4, '["shaved chocolate","white chocolate"]', 338), (8, '["vanilla foam","marshmallow"]', 263), (1, '["peppermint","crispy rice"]', 205), (6, '["shaved chocolate","dark chocolate","crispy rice","cinnamon","peppermint"]', 161), (7, '["caramel drizzle","crispy rice","marshmallow","vanilla foam","cinnamon"]', 132), (3, '["vanilla foam","peppermint"]', 95); ``` Changing `TEXT[]` to `TEXT` is not required, as SQLITE really doesn't bother with the types unless the table is strict. We'll take that flexibility here. ## Problem > Get the stewards a list of all the passengers and the cocoa car(s) they can be served from that has at least one of their favorite mixins. > > Remember only the top three most-stocked cocoa cars remained operational, so the passengers must be served from one of those cars. Ok so, we have two tables. 1. `cocoa_cars` 2. `passengers` What we need to do is to list down the passengers which the `cocoa_cars` can satisfy their atleast one mixin(food) with their car_ids. So, there are limited cars, just 9 cars in total right? ```sql SELECT * FROM cocoa_cars ORDER BY car_id; ``` So, there are only 9 cars. ``` sqlite> SELECT * FROM cocoa_cars ORDER BY car_id; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` However, the question states, we only need to consider the top three most stacked cars. Now we can order by `total_stack` here and find the top stacked 3 cars with the `LIMIT 3` clause. ```sql SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3; ``` ``` sqlite> sqlite> select * from cocoa_cars ORDER BY total_stock DESC; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ | 4 | ["shaved chocolate","white chocolate"] | 338 | +--------+--------------------------------------------------------------+-------------+ | 8 | ["vanilla foam","marshmallow"] | 263 | +--------+--------------------------------------------------------------+-------------+ | 1 | ["peppermint","crispy rice"] | 205 | +--------+--------------------------------------------------------------+-------------+ | 6 | ["shaved chocolate","dark chocolate","crispy rice","cinnamon | 161 | | | ","peppermint"] | | +--------+--------------------------------------------------------------+-------------+ | 7 | ["caramel drizzle","crispy rice","marshmallow","vanilla foam | 132 | | | ","cinnamon"] | | +--------+--------------------------------------------------------------+-------------+ | 3 | ["vanilla foam","peppermint"] | 95 | +--------+--------------------------------------------------------------+-------------+ sqlite> select * from cocoa_cars ORDER BY total_stock DESC LIMIT 3; +--------+--------------------------------------------------------------+-------------+ | car_id | available_mixins | total_stock | +--------+--------------------------------------------------------------+-------------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | +--------+--------------------------------------------------------------+-------------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | +--------+--------------------------------------------------------------+-------------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | | | ate"] | | +--------+--------------------------------------------------------------+-------------+ sqlite> ``` Now, we have all the cars that we can start assigning the passengers. How? We need to select and assign the passenger the `car_id` which contains one or more of their `favourite_mixins`. Now, this is the tricky part. We are in SQLite! We have a `favourite_list` for `Mateo Cruz` as a string like `["caramel drizzle","shaved chocolate","white chocolate"]` which we need to match against these 3 car `available_mixins`: - `["white chocolate","shaved chocolate"]` on car_id `5` - `["cinnamon","marshmallow","caramel drizzle"]` on car_id `2` - `["crispy rice","peppermint","caramel drizzle","shaved chocolate"]` on car_id `9` So here all three cars have at least one right? - car_id `5` has `shaved_chocolate` - car_id `2` has `caramel drizzle` - car_id `9` has 2 of his 3 `favourite_list` dishes `caramel drizzle`, and `shaved chocolate"`. So we should ideally return for `Matro Cruz` the car_ids `[5, 2, 9]` or as separate rows doesn't matter as much I think. But the first one looks cool! So, how do we do it? First let's start with what we had! The top 3 cars as ```sql SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3; ``` This gives us the table that we can use to compare for each car_id if we have any one of the `favourite_list` for one passenger at a time. But how do we split the string of mixins and favourite_list? We can use the [json_each](https://sqlite.org/json1.html#jeach) function which takes in any valid json string (could be raw, could be a column name) for each row of the the table. And it returns back a lot of things. Let's just try to select everything from the `json_each` with the favourite_mixins column. ```sql SELECT * FROM json_each(favorite_mixins) FROM passengers; ``` ```sqlite> SELECT * FROM json_each(favorite_mixins) FROM passengers; Parse error: near "FROM": syntax error SELECT * FROM json_each(favorite_mixins) FROM passengers; error here ---^ sqlite> ``` Ops! Why? Because we have given it the whole column, it can only take one cell at a time, so we need to give it that cell for each row. ```sql SELECT * FROM json_each((SELECT favorite_mixins FROM passengers)); ``` Here we try to give it only the `favorite_mixins` from the passengers column. ``` sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers)); +-----+--------------+------+--------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+--------------+------+--------------+----+--------+---------+------+ | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +-----+--------------+------+--------------+----+--------+---------+------+ sqlite> select * from passengers limit 5; +--------------+----------------+--------------------------------------------------------------+--------+ | passenger_id | passenger_name | favorite_mixins | car_id | +--------------+----------------+--------------------------------------------------------------+--------+ | 1 | Ava Johnson | ["vanilla foam"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | +--------------+----------------+--------------------------------------------------------------+--------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | +--------------+----------------+--------------------------------------------------------------+--------+ | 5 | Layla Brooks | ["crispy rice","dark chocolate","caramel drizzle","cinnamon" | 3 | | | | ] | | +--------------+----------------+--------------------------------------------------------------+--------+ sqlite> ``` But it only gave it for the first column, unluckily it had just one mixin. What is happening? Well! This actually selects only the first row's `favorite_mixins` because `json_each()` processes one value at a time. Since favorite_mixins is a JSON array, SQLite expects a single array value per row. When we try to pass the entire column, it only processes the first row of `favorite_mixins` Let's try to use the json_each for passengers with more than one favorite mixins. ```sql SELECT * FROM json_each( (SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Mateo Cruz') ); ``` ``` sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Ava Johnson')); +-----+--------------+------+--------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+--------------+------+--------------+----+--------+---------+------+ | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +-----+--------------+------+--------------+----+--------+---------+------+ sqlite> SELECT * FROM json_each((SELECT favorite_mixins FROM passengers WHERE passenger_name = 'Mateo Cruz')); +-----+------------------+------+------------------+----+--------+---------+------+ | key | value | type | atom | id | parent | fullkey | path | +-----+------------------+------+------------------+----+--------+---------+------+ | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | +-----+------------------+------+------------------+----+--------+---------+------+ sqlite> ``` Ok now that is neat, it returned 3 rows for the 3 favorite items for the passenger `Mateo Cruz` Now what? How do we get it for all the passengers? How about JOINs, since the passenger's data will remain the same, we just change the mixins for each of their favorite list. ```sql SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; ``` ``` sqlite> sqlite> SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | passenger_id | passenger_name | favorite_mixins | car_id | key | value | type | atom | id | parent | fullkey | path | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 127 | Aaron Ashcroft | ["dark chocolate","marshmallow"] | 4 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 127 | Aaron Ashcroft | ["dark chocolate","marshmallow"] | 4 | 1 | marshmallow | text | marshmallow | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 104 | Abigail Worthington | ["dark chocolate","caramel drizzle"] | 8 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 104 | Abigail Worthington | ["dark chocolate","caramel drizzle"] | 8 | 1 | caramel drizzle | text | caramel drizzle | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 157 | Aiden Stanfield | ["peppermint","crispy rice"] | 8 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 157 | Aiden Stanfield | ["peppermint","crispy rice"] | 8 | 1 | crispy rice | text | crispy rice | 13 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | | | | late"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 1 | shaved chocolate | text | shaved chocolate | 13 | | $[1] | $ | | | | late"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 103 | Alexander Stratton | ["peppermint","shaved chocolate","vanilla foam","white choco | 8 | 2 | vanilla foam | text | vanilla foam | 31 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ ... ... | 153 | Wyatt Alderton | ["white chocolate"] | 3 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 15 | Yara Haddad | ["white chocolate","dark chocolate"] | 2 | 1 | dark chocolate | text | dark chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 51 | Youssef El-Sayed | ["peppermint","marshmallow"] | 3 | 0 | peppermint | text | peppermint | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 51 | Youssef El-Sayed | ["peppermint","marshmallow"] | 3 | 1 | marshmallow | text | marshmallow | 13 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 0 | dark chocolate | text | dark chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 1 | caramel drizzle | text | caramel drizzle | 18 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 57 | Zachary Collins | ["dark chocolate","caramel drizzle","vanilla foam"] | 2 | 2 | vanilla foam | text | vanilla foam | 35 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 131 | Zachary Stafford | ["shaved chocolate"] | 2 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 0 | vanilla foam | text | vanilla foam | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 1 | crispy rice | text | crispy rice | 16 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 11 | Zara Sheikh | ["vanilla foam","crispy rice","peppermint"] | 4 | 2 | peppermint | text | peppermint | 28 | | $[2] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 42 | Zoe Wilson | ["marshmallow","dark chocolate"] | 9 | 0 | marshmallow | text | marshmallow | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ | 42 | Zoe Wilson | ["marshmallow","dark chocolate"] | 9 | 1 | dark chocolate | text | dark chocolate | 14 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+ ``` It just automagically joins the relevant rows for the expanded rows from the json_each when we pass the column which is a json list. But we don't want everything, we just want the passenger and the mixin names. And let's also include the index just to see the data. ```sql SELECT passengers.passenger_name, mixin.key AS mixin_index, mixin.value AS mixin FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; ``` ``` sqlite> SELECT passengers.passenger_name, mixin.key AS mixin_index, mixin.value AS mixin FROM passengers JOIN json_each(passengers.favorite_mixins) AS mixin ORDER BY passengers.passenger_name; +------------------------+-------------+------------------+ | passenger_name | mixin_index | mixin | +------------------------+-------------+------------------+ | Aaron Ashcroft | 0 | dark chocolate | | Aaron Ashcroft | 1 | marshmallow | | Abigail Worthington | 0 | dark chocolate | | Abigail Worthington | 1 | caramel drizzle | | Aiden Stanfield | 0 | peppermint | | Aiden Stanfield | 1 | crispy rice | | Alexander Stratton | 0 | peppermint | | Alexander Stratton | 1 | shaved chocolate | | Alexander Stratton | 2 | vanilla foam | | Alexander Stratton | 3 | white chocolate | ... ... | Yara Haddad | 1 | dark chocolate | | Youssef El-Sayed | 0 | peppermint | | Youssef El-Sayed | 1 | marshmallow | | Zachary Collins | 0 | dark chocolate | | Zachary Collins | 1 | caramel drizzle | | Zachary Collins | 2 | vanilla foam | | Zachary Stafford | 0 | shaved chocolate | | Zara Sheikh | 0 | vanilla foam | | Zara Sheikh | 1 | crispy rice | | Zara Sheikh | 2 | peppermint | | Zoe Wilson | 0 | marshmallow | | Zoe Wilson | 1 | dark chocolate | +------------------------+-------------+------------------+ ``` That looks good. Now what? We also need to do it for the cocoa_cars with `available_mixins` ```sql SELECT * FROM cocoa_cars JOIN json_each(available_mixins) AS mixin ORDER BY total_stock DESC; ``` OR ```sql SELECT car_id, mixin.key AS mixin_index, mixin.value AS mixin FROM cocoa_cars JOIN json_each(available_mixins) as mixin ORDER BY total_stock DESC; ``` ``` sqlite> SELECT car_id, mixin.key as mixin_index, mixin.value as mixin from cocoa_cars join json_each(available_mixins) as mixin ORDER BY total_stock DESC; +--------+-------------+------------------+ | car_id | mixin_index | mixin | +--------+-------------+------------------+ | 5 | 0 | white chocolate | | 5 | 1 | shaved chocolate | | 2 | 0 | cinnamon | | 2 | 1 | marshmallow | | 2 | 2 | caramel drizzle | | 9 | 0 | crispy rice | | 9 | 1 | peppermint | | 9 | 2 | caramel drizzle | | 9 | 3 | shaved chocolate | | 4 | 0 | shaved chocolate | | 4 | 1 | white chocolate | | 8 | 0 | vanilla foam | | 8 | 1 | marshmallow | | 1 | 0 | peppermint | | 1 | 1 | crispy rice | | 6 | 0 | shaved chocolate | | 6 | 1 | dark chocolate | | 6 | 2 | crispy rice | | 6 | 3 | cinnamon | | 6 | 4 | peppermint | | 7 | 0 | caramel drizzle | | 7 | 1 | crispy rice | | 7 | 2 | marshmallow | | 7 | 3 | vanilla foam | | 7 | 4 | cinnamon | | 3 | 0 | vanilla foam | | 3 | 1 | peppermint | +--------+-------------+------------------+ sqlite> ``` But hold on! We only needed it for the top 3 stocked cars. So, how do we do it, we can (bad and dirty practise) limit by the count of the numbers of mixin in 5, 2 and 9 but that is bad. We need to dynamically get this table. Help in! CTEs ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM stocked_cars JOIN json_each(stocked_cars.available_mixins); ``` OR ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT car_id, car_mixins.key as car_mixin_index, car_mixins.value as car_mixin FROM stocked_cars JOIN json_each(stocked_cars.available_mixins) as car_mixins; ``` ``` sqlite> WITH stocked_cars as ( (x1...> select * from cocoa_cars ORDER BY total_stock DESC LIMIT 3) ...> select * from stocked_cars JOIN json_each(stocked_cars.available_mixins); +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | car_id | available_mixins | total_stock | key | value | type | atom | id | parent | fullkey | path | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 0 | cinnamon | text | cinnamon | 2 | | $[0] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 1 | marshmallow | text | marshmallow | 11 | | $[1] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | ["cinnamon","marshmallow","caramel drizzle"] | 359 | 2 | caramel drizzle | text | caramel drizzle | 23 | | $[2] | $ | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 3 | shaved chocolate | text | shaved chocolate | 42 | | $[3] | $ | | | ate"] | | | | | | | | | | +--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT car_id, car_mixins.key as car_mixin_index, car_mixins.value as car_mixin FROM stocked_cars JOIN json_each(stocked_cars.available_mixins) as car_mixins; +--------+-----------------+------------------+ | car_id | car_mixin_index | car_mixin | +--------+-----------------+------------------+ | 5 | 0 | white chocolate | | 5 | 1 | shaved chocolate | | 2 | 0 | cinnamon | | 2 | 1 | marshmallow | | 2 | 2 | caramel drizzle | | 9 | 0 | crispy rice | | 9 | 1 | peppermint | | 9 | 2 | caramel drizzle | | 9 | 3 | shaved chocolate | +--------+-----------------+------------------+ sqlite> ``` Now what next? We simply need to combine all of it - Grab the top 3 cars - Grab the passenger favorite mixins (expand with json_each) - Grab the car available mixins (expand with json_each and use the top 3 car as cte) - JOIN them when the car_mixin has one mixin from a passenger favorite mixin. ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` OR Select some more rows for visual confirmation. ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, passengers.favorite_mixins, stocked_cars.available_mixins, passenger_mixin.value as passenger_mixin, available_mixin.value as available_mixin, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` OR select everything ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; ``` ``` sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +------------------------+--------+ | passenger_name | car_id | +------------------------+--------+ | Mateo Cruz | 5 | | Mateo Cruz | 5 | | Nia Grant | 5 | | Hiro Tanaka | 5 | | Ravi Patel | 5 | | Ravi Patel | 5 | | Elena Morales | 5 | | Elena Morales | 5 | | Diego Ramos | 5 | | Caleb Osei | 5 | | Caleb Osei | 5 | | Lucas Ford | 5 | | Yara Haddad | 5 | | Tariq Hassan | 5 | | Eva Schmidt | 5 | | Ingrid Nilsen | 5 | | Sophia Rossi | 5 | | Olivia Dubois | 5 | | Emma Svensson | 5 | | Isabella Laurent | 5 | | James Kim | 5 | ... ... | Griffin Hartwell | 9 | | Griffin Hartwell | 9 | | Megan Redfield | 9 | | Megan Redfield | 9 | | Grayson Westmore | 9 | | Nicole Ashridge | 9 | | Sawyer Hollingsworth | 9 | | Sawyer Hollingsworth | 9 | | Alexis Thorndale | 9 | | Declan Summerfield | 9 | | Declan Summerfield | 9 | | Samantha Brightwood | 9 | | Samantha Brightwood | 9 | | Tristan Ashbrook | 9 | | Lauren Silverton | 9 | | Landon Whitworth | 9 | | Kayla Mansfield | 9 | | Kayla Mansfield | 9 | +------------------------+--------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, passengers.favorite_mixins, stocked_cars.available_mixins, passenger_mixin.value as passenger_mixin, available_mixin.value as available_mixin, stocked_cars.car_id FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | passenger_name | favorite_mixins | available_mixins | passenger_mixin | available_mixin | car_id | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Nia Grant | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Hiro Tanaka | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Diego Ramos | ["shaved chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | shaved chocolate | shaved chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Caleb Osei | ["shaved chocolate","dark chocolate","white chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Lucas Ford | ["vanilla foam","white chocolate","cinnamon"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Yara Haddad | ["white chocolate","dark chocolate"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Tariq Hassan | ["dark chocolate","crispy rice","white chocolate","peppermin | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | | | t"] | | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Eva Schmidt | ["white chocolate","marshmallow"] | ["white chocolate","shaved chocolate"] | white chocolate | white chocolate | 5 | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ ... ... +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Alexis Thorndale | ["vanilla foam","crispy rice","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Tristan Ashbrook | ["crispy rice"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | crispy rice | crispy rice | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Lauren Silverton | ["caramel drizzle","vanilla foam","cinnamon","dark chocolate | ["crispy rice","peppermint","caramel drizzle","shaved chocol | caramel drizzle | caramel drizzle | 9 | | | "] | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Landon Whitworth | ["caramel drizzle","vanilla foam"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | caramel drizzle | caramel drizzle | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | peppermint | peppermint | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | ["crispy rice","peppermint","caramel drizzle","shaved chocol | shaved chocolate | shaved chocolate | 9 | | | | ate"] | | | | +------------------------+--------------------------------------------------------------+--------------------------------------------------------------+------------------+------------------+--------+ sqlite> sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT * FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value; +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | passenger_id | passenger_name | favorite_mixins | car_id | key | value | type | atom | id | parent | fullkey | path | car_id | available_mixins | total_stock | key | value | type | atom | id | parent | fullkey | path | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 2 | Mateo Cruz | ["caramel drizzle","shaved chocolate","white chocolate"] | 2 | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 3 | Nia Grant | ["shaved chocolate"] | 5 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 4 | Hiro Tanaka | ["shaved chocolate"] | 2 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 6 | Ravi Patel | ["caramel drizzle","shaved chocolate","white chocolate"] | 5 | 2 | white chocolate | text | white chocolate | 37 | | $[2] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 0 | white chocolate | text | white chocolate | 2 | | $[0] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 9 | Elena Morales | ["white chocolate","shaved chocolate","caramel drizzle"] | 6 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 10 | Diego Ramos | ["shaved chocolate"] | 1 | 0 | shaved chocolate | text | shaved chocolate | 2 | | $[0] | $ | 5 | ["white chocolate","shaved chocolate"] | 412 | 1 | shaved chocolate | text | shaved chocolate | 19 | | $[1] | $ | ... ... +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 171 | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | 8 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 171 | Sawyer Hollingsworth | ["crispy rice","dark chocolate","peppermint"] | 8 | 2 | peppermint | text | peppermint | 30 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 172 | Alexis Thorndale | ["vanilla foam","crispy rice","dark chocolate"] | 9 | 1 | crispy rice | text | crispy rice | 16 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 173 | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | 7 | 2 | crispy rice | text | crispy rice | 30 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 173 | Declan Summerfield | ["dark chocolate","marshmallow","crispy rice","peppermint"] | 7 | 3 | peppermint | text | peppermint | 42 | | $[3] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 174 | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | 4 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 174 | Samantha Brightwood | ["crispy rice","peppermint","dark chocolate"] | 4 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 175 | Tristan Ashbrook | ["crispy rice"] | 1 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 0 | crispy rice | text | crispy rice | 2 | | $[0] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 178 | Lauren Silverton | ["caramel drizzle","vanilla foam","cinnamon","dark chocolate | 2 | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | | "] | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 179 | Landon Whitworth | ["caramel drizzle","vanilla foam"] | 8 | 0 | caramel drizzle | text | caramel drizzle | 2 | | $[0] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 2 | caramel drizzle | text | caramel drizzle | 25 | | $[2] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 180 | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | 2 | 1 | peppermint | text | peppermint | 16 | | $[1] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 1 | peppermint | text | peppermint | 14 | | $[1] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ | 180 | Kayla Mansfield | ["vanilla foam","peppermint","shaved chocolate"] | 2 | 2 | shaved chocolate | text | shaved chocolate | 27 | | $[2] | $ | 9 | ["crispy rice","peppermint","caramel drizzle","shaved chocol | 354 | 3 | shaved chocolate | text | shaved chocolate | 42 | | $[3] | $ | | | | | | | | | | | | | | | ate"] | | | | | | | | | | +--------------+------------------------+--------------------------------------------------------------+--------+-----+------------------+------+------------------+----+--------+---------+------+--------+--------------------------------------------------------------+-------------+-----+------------------+------+------------------+----+--------+---------+------+ sqlite> ``` Now what we did here? - Create the top 3 stocked cars as CTE - The main query first fetches each passenger with its favorite mixin expanded - It joined the table of available_mixin when? - The mixin from passenger is equal to the mixin in the available mixin in the car - Hence we get the car_id as the mixin served for that passenger. You can see we have multiple rows for each user, we might not want that. Though nothing is wrong with it, but the report looks quite long, especially if there were more than a couple of mixins for the passenger or the number of cars were more. We can group by the `passenger_name` and sort of concatenate the ```sql WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' AS cocoa_cars FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value GROUP BY passengers.passenger_name; ``` Here, we have added, ``` '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' ``` AND to group by ``` GROUP BY passengers.passenger_name; ``` So this will squish down all the separate rows of the passenger favorite mixins and the cars that have it, with a single row and we defined how we want to squish the different `car_id`s with a [GROUP_CONCAT](https://www.sqlite.org/lang_aggfunc.html#group_concat). This function can concatentate (join or attach or combine) together the multiple strings with a specific separator (by default the separate is `,`). We also use the `||` concatenation operator to add `[` at the start and `]` at the end of the list of `car_ids`. > The group_concat() function returns a string which is the concatenation of all non-NULL values of X. If parameter Y is present then it is used as the separator between instances of X. A comma (",") is used as the separator if Y is omitted. ``` sqlite> WITH stocked_cars as ( SELECT * FROM cocoa_cars ORDER BY total_stock DESC LIMIT 3 ) SELECT passengers.passenger_name, '[' || GROUP_CONCAT(DISTINCT stocked_cars.car_id) || ']' AS cocoa_cars FROM passengers JOIN json_each(passengers.favorite_mixins) as passenger_mixin JOIN stocked_cars JOIN json_each(stocked_cars.available_mixins) AS available_mixin ON passenger_mixin.value = available_mixin.value ...> GROUP BY passengers.passenger_name; +------------------------+------------+ | passenger_name | cocoa_cars | +------------------------+------------+ | Aaron Ashcroft | [2] | | Abigail Worthington | [2,9] | | Aiden Stanfield | [9] | | Alexander Stratton | [5,9] | | Alexis Thorndale | [9] | | Alice Merriweather | [5,2,9] | | Amanda Fitzroy | [5,2,9] | | Amelia Rosewood | [9] | | Anna Westfield | [2] | | Aria Blackwood | [5,2,9] | | Ashley Fairhaven | [5,2,9] | | August Blackwell | [5,2,9] | | Aurora Whitfield | [5,2,9] | | Benjamin Fairweather | [9] | | Benjamin Patel | [9] | | Bianca Pereira | [9] | | Blake Dunwood | [5,9] | | Brandon Locklear | [5,2,9] | | Caleb Osei | [5,9] | | Cameron Gladstone | [2,9] | | Cameron Rogers | [5,2,9] | | Carlos Mendez | [5,9] | | Caroline Bannister | [2,9] | | Carter Brookfield | [2] | | Charlotte Singh | [5,2,9] | | Charlotte Waverly | [5,9] | | Chase Ashbury | [5,9] | | Chloe Blackstone | [5,9] | | Claire Berkshire | [5,9] | | Clara Westbrook | [2] | | Cole Ashland | [2] | | Connor Redmond | [2,9] | | Daniel Murphy | [5,2] | | Declan Summerfield | [2,9] | | Diana Brightwell | [5,9] | | Diego Ramos | [5,9] | | Dylan Blakely | [5,2,9] | | Eleanor Morrison | [5,9] | | Elena Morales | [5,2,9] | | Elijah Wakefield | [2,9] | | Eliza Radcliffe | [5,9] | | Elizabeth Chesterfield | [5,2,9] | | Emily Claridge | [5,9] | | Emily Johnson | [2] | | Emma Gilmore | [5,2,9] | | Emma Svensson | [5,9] | | Ethan Marlowe | [5,2,9] | | Eva Schmidt | [5,2] | | Eva Templeton | [5,2,9] | | Fatima Noor | [9] | | Felix Schneider | [9] | | Felix Whitmore | [2] | | Finn Lockhart | [2,9] | | Gabriel Winthrop | [5,2,9] | | Grace Aldridge | [5,2,9] | | Grace Thornhill | [5,2,9] | | Grayson Westmore | [2,9] | | Griffin Hartwell | [9] | | Hannah Livingston | [5,2,9] | | Hazel Kincaid | [2,9] | | Henry Treadwell | [2,9] | | Hiro Tanaka | [5,9] | | Hunter Bellingham | [5,9] | | Ian Landsman | [2] | | Ingrid Nilsen | [5,2,9] | | Iris Pembroke | [5,2,9] | | Isaac Kendrick | [2] | | Isabella Laurent | [5,9] | | Isabella Rodriguez | [9] | | Ivy Winslow | [2,9] | | Jack Carmichael | [9] | | Jackson Thorpe | [2,9] | | James Garrison | [5,9] | | James Kim | [5,2,9] | | James Simon | [5,2,9] | | Jason Stewart | [5] | | Jasper Thorne | [5,2,9] | | Jessica Langston | [5,2,9] | | Jonah Wolfe | [2] | | Jordan Waverly | [5,9] | | Julia Pendleton | [5,2,9] | | Julian Blackburn | [5] | | Kayla Mansfield | [5,9] | | Keiko Ito | [2] | | Landon Whitworth | [2,9] | | Laura Thornbury | [5,2,9] | | Lauren Silverton | [2,9] | | Layla Brooks | [2,9] | | Leo Greyson | [2,9] | | Liam Donovan | [2,9] | | Liam OConnor | [2,9] | | Linda Lee | [5,2] | | Logan Sherwood | [5,9] | | Lucas Ford | [5,2] | | Lucas Prescott | [9] | | Lucy Morris | [2,9] | | Luna Hartley | [5] | | Madeline Ramsey | [2,9] | | Madison Clearwater | [5,2,9] | | Margaret Fairbanks | [2,9] | | Mason Fairmont | [2,9] | | Mateo Cruz | [5,2,9] | | Matthew Sutherland | [2,9] | | Megan Redfield | [5,9] | | Melissa Ravenscroft | [5] | | Mia Chen | [5,9] | | Mila Novak | [2,9] | | Miles Brennan | [5,2,9] | | Milo Ashford | [2,9] | | Mira Zhao | [2,9] | | Morgan Steadman | [5,9] | | Natalie Warwick | [2] | | Nathan Whitley | [5,9] | | Nia Grant | [5,9] | | Nicholas Montague | [5] | | Nicole Ashridge | [9] | | Noah Fischer | [2,9] | | Nolan Murphy | [2,9] | | Nolan Young | [5,2,9] | | Nora Calloway | [9] | | Nova Adams | [2] | | Oliver Ashby | [9] | | Olivia Carrington | [2] | | Olivia Dubois | [5,9] | | Omar Qureshi | [2] | | Owen Fairchild | [5,9] | | Parker Blackmore | [5,2,9] | | Penelope Sinclair | [5,2,9] | | Rachel Wyndham | [5,9] | | Rafael Silva | [2,9] | | Ravi Patel | [5,2,9] | | Rebecca Ashcombe | [2,9] | | Robert Smith | [9] | | Rose Drummond | [5,2,9] | | Ruby Hawthorne | [2] | | Ryan Beckett | [5,9] | | Samantha Brightwood | [9] | | Samuel Kingsley | [5,2,9] | | Sara Johansson | [2] | | Sarah Davis | [9] | | Sarah Remington | [5,2,9] | | Sawyer Hollingsworth | [9] | | Scarlett Dalton | [5,2,9] | | Silas Merrick | [2] | | Sofia Kim | [2] | | Sophia Brookshire | [2,9] | | Sophia Rossi | [5,2,9] | | Sophie Langford | [5,9] | | Stella Beaumont | [2,9] | | Tariq Hassan | [5,9] | | Taylor Woodridge | [5,2,9] | | Theodore Hadley | [2] | | Tim Cook | [9] | | Tristan Ashbrook | [9] | | Victoria Harwood | [2,9] | | Violet Sterling | [2,9] | | William Beauchamp | [5,2,9] | | William Becker | [9] | | William Chen | [5,2,9] | | Wyatt Alderton | [5] | | Yara Haddad | [5] | | Youssef El-Sayed | [2,9] | | Zachary Collins | [2,9] | | Zachary Stafford | [5,9] | | Zara Sheikh | [9] | | Zoe Wilson | [2] | +------------------------+------------+ sqlite> ``` I think we are done. We did it, it was a bit different one. Some wired hack here and there but we made it! Day 7 done and dusted. On to the day 8. sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-6 Advent of SQL Day 6: Days of Delight It is day 6 of advent of SQL. Let's jump straight into the sql for the day. So, we have two tables: table table The first t Sun, 21 Dec 2025 00:00:00 UTC ## Advent of SQL Day 6: Days of Delight It is day 6 of advent of SQL. Let's jump straight into the sql for the day. ```sql DROP TABLE IF EXISTS families; DROP TABLE IF EXISTS deliveries_assigned; CREATE TABLE families ( id INT PRIMARY KEY, family_name TEXT ); CREATE TABLE deliveries_assigned ( id INT PRIMARY KEY, family_id INT, gift_date DATE, gift_name TEXT ); INSERT INTO families (id, family_name) VALUES (1, 'Isla Martinez'), (2, 'Nolan Garcia'), (3, 'Yara Chen'), (4, 'Tariq Nguyen'), (5, 'Mila Hernandez'), (6, 'Casey Kim'), (7, 'Mateo Hernandez'), (8, 'Keiko Petrov'), (9, 'Ethan Flores'), (10, 'Mateo Nakamura'), (11, 'Maya Fernandez'), (12, 'Mila Davis'), (13, 'Yara Rossi'), (14, 'Nolan Phillips'), (15, 'Amina Perez'); INSERT INTO deliveries_assigned (id, family_id, gift_date, gift_name) VALUES (1, 1, '2025-12-01', 'roasted cashews'), (2, 1, '2025-12-02', 'cookie decorating kit'), (3, 1, '2025-12-03', 'dark chocolate assortment'), (4, 1, '2025-12-04', 'white chocolate candies'), (5, 1, '2025-12-05', 'reindeer headband'), (6, 1, '2025-12-06', 'holiday brownie bites'), (7, 1, '2025-12-07', 'shortbread cookie tin'), (8, 1, '2025-12-08', 'chocolate chip cookies'), (9, 1, '2025-12-11', 'holiday jam trio'), (10, 1, '2025-12-12', 'white chocolate popcorn'), (11, 1, '2025-12-14', 'holiday jam trio'), (12, 1, '2025-12-15', 'fudge bites'), (13, 1, '2025-12-16', 'holiday sticker sheet'), (14, 1, '2025-12-18', 'hot cocoa bombs'), (15, 1, '2025-12-19', 'honey roasted nuts'), (16, 1, '2025-12-20', 'holiday mug'), (17, 1, '2025-12-21', 'white chocolate candies'), (18, 1, '2025-12-22', 'puzzle book'), (19, 1, '2025-12-23', 'snowman plush'), (20, 1, '2025-12-24', 'scented hand cream'), (21, 1, '2025-12-25', 'vanilla bean wafers'), (22, 2, '2025-12-01', 'roasted cashews'), (23, 2, '2025-12-02', 'holiday brownie bites'), (24, 2, '2025-12-03', 'peppermint bark bites'), (25, 2, '2025-12-04', 'holiday jam trio'), (26, 2, '2025-12-05', 'festive notepad'), (27, 2, '2025-12-06', 'scented pine sachet'), (28, 2, '2025-12-07', 'holiday mug'), (29, 2, '2025-12-08', 'shortbread cookie tin'), (30, 2, '2025-12-09', 'dark chocolate assortment'); ``` So, we have two tables: 1. `families` table 2. `deliveries_assigned` table The first table, `family` just has the id and the name of the family. The second table, `deliveries_assigned` has the id, family id, gift date, and gift name. Let's look at the problem statement. ## Problem > Generate a report that returns the dates and families that have no delivery assigned after December 14th, using the `families` and `deliveries_assigned`. > > Each row in the report should be a date and family name that represents the dates in which families don't have a delivery assigned yet. > > Label the columns as `unassigned_date` and `name`. Order the results by `unassigned_date` and `name`, respectively, both in ascending order. Ok, so we need to list the deliveries_assigned for each family first, to check what kind of pattern are we looking for. ```sql SELECT * FROM deliveries_assigned WHERE family_id = 1; ``` ``` sqlite> select * from deliveries_assigned where family_id=1; +----+-----------+------------+---------------------------+ | id | family_id | gift_date | gift_name | +----+-----------+------------+---------------------------+ | 1 | 1 | 2025-12-01 | roasted cashews | | 2 | 1 | 2025-12-02 | cookie decorating kit | | 3 | 1 | 2025-12-03 | dark chocolate assortment | | 4 | 1 | 2025-12-04 | white chocolate candies | | 5 | 1 | 2025-12-05 | reindeer headband | | 6 | 1 | 2025-12-06 | holiday brownie bites | | 7 | 1 | 2025-12-07 | shortbread cookie tin | | 8 | 1 | 2025-12-08 | chocolate chip cookies | | 9 | 1 | 2025-12-11 | holiday jam trio | | 10 | 1 | 2025-12-12 | white chocolate popcorn | | 11 | 1 | 2025-12-14 | holiday jam trio | | 12 | 1 | 2025-12-15 | fudge bites | | 13 | 1 | 2025-12-16 | holiday sticker sheet | | 14 | 1 | 2025-12-18 | hot cocoa bombs | | 15 | 1 | 2025-12-19 | honey roasted nuts | | 16 | 1 | 2025-12-20 | holiday mug | | 17 | 1 | 2025-12-21 | white chocolate candies | | 18 | 1 | 2025-12-22 | puzzle book | | 19 | 1 | 2025-12-23 | snowman plush | | 20 | 1 | 2025-12-24 | scented hand cream | | 21 | 1 | 2025-12-25 | vanilla bean wafers | +----+-----------+------------+---------------------------+ sqlite> ``` So, we are missing gifts for family id `1` on `09`, `10`, `13`, and `17`. But we are only asked for gifts after `14` (December 14th). > Generate a report that returns the dates and families that have **no delivery assigned after December 14th**, using the families and deliveries_assigned. So, we can discard `09`, `10`, and `13` as they are before December 14th. ```sql SELECT * FROM deliveries_assigned WHERE family_id = 1 AND gift_date > '2025-12-14'; ``` It gives the right dates where the gifts are assigned after December 14th. But the problem is we need to get dates which are missing in this `deliveries_assigned` table record for each family. Finding something missing is kind of wired, because you don't have what is missing. Especially for dates, like dates are very painful. We need to find among the sequential order of the dates, when some of the dates are missing, that is simple here, but you can see it could be quite cubersome if we have to manually add each date in the list for comparing with. ### JOINs with NOT EXISTS So, the basic dirty solution is to check the missing dates for each family, one by one. ```sql SELECT families.family_name AS name, dates.column1 AS unassigned_date FROM families JOIN ( VALUES ('2025-12-15'), ('2025-12-16'), ('2025-12-17'), ('2025-12-18'), ('2025-12-19'), ('2025-12-20'), ('2025-12-21'), ('2025-12-22'), ('2025-12-23'), ('2025-12-24'), ('2025-12-25') ) AS dates ON 1=1 WHERE NOT EXISTS ( SELECT 1 FROM deliveries_assigned WHERE deliveries_assigned.family_id = families.id AND deliveries_assigned.gift_date = dates.column1 ) ORDER BY unassigned_date, name; ``` Let's break it down: 1. We have a `families` table with id and family name. 2. We have a `deliveries_assigned` table with id, family id, gift date, and gift name. 3. We create a list of dates from `2025-12-15` to `2025-12-25` using the `VALUES` keyword. - This just appends one date after other and names the columns as `column1` with the table as `dates`. - The `ON` condition is `1=1` to make sure the `WHERE` condition is true for the JOIN to happen. 4. We use the `NOT EXISTS` keyword to check if the `deliveries_assigned` table has a record for each date in the list. - we use `NOT EXISTS` because we want to check if there is no date for each date in the assigned list. 5. We order the results by `unassigned_date` and `name`, respectively, both in ascending order. So, that is not the best way to solve this, I think. ### Recursive CTEs We can generate a table full of dates, and then cross join the table with the `families` table. This will give us all possible combinations of dates and families like the gift should ideally be there for each family from 1st to 25th of December(however we are only interested from 15th to 25th December). Then once we have that full table of combination, we can check from the `deliveries_assigned` table and inner join for each family and filter out the rows which have family_id as `NULL` because some dates will be missing for that family. First we'll create a recursive table of dates from `2025-12-15` to `2025-12-25` ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT * FROM dates; ``` This will give us the dates from `2025-12-15` to `2025-12-25` in a recursive table. What is a recursive table? > A recursive table is a table that is defined as a combination of itself. So, we can create a recursive table of dates from `2025-12-15` to `2025-12-25`, The base case is `2025-12-15`, and the recursive case is `SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25'` which means it will call it recursively for `2025-12-16` on the first call inside it because of `+1 day` as the interval in the `date` function. The date function is a [function](https://sqlite.org/lang_datefunc.html#modifiers) which takes in a date and we can add modifiers to it to manipulate or extract parts of the date. Here we have added the modifier as `+1 day` which will increment the day by one. We then call that in the `dates` CTE again till we have the `date` as less then `2025-12-25`. Till then we will have created all the `dates` from `15` till `25` including `25`. We can just type in the values manually as we did in the first dirty solution, but I wanted to see how we can generate dates dynamically in sqlite. ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT * FROM dates; +------------+ | gift_date | +------------+ | 2025-12-15 | | 2025-12-16 | | 2025-12-17 | | 2025-12-18 | | 2025-12-19 | | 2025-12-20 | | 2025-12-21 | | 2025-12-22 | | 2025-12-23 | | 2025-12-24 | | 2025-12-25 | +------------+ Run Time: real 0.000 user 0.000225 sys 0.000009 sqlite> ``` Now we'll cross join the `families` table with the `dates` table. ```sql SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates; ``` OK! Wait include the `dates` CTE above too, was just simplifying the query. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates; ``` This will give us the full table of combination of dates and families. ```sql SELECT COUNT(*) FROM families; ``` ```sql sqlite> SELECT COUNT(*) FROM families; +----------+ | COUNT(*) | +----------+ | 250 | +----------+ Run Time: real 0.000 user 0.000113 sys 0.000006 sqlite> ``` That is : - There are 11 dates right? `2025-12-15` to `2025-12-25`, -> `15` (1), `16` (2), `17` (3), `18` (4), `19` (5), `20` (6), `21` (7), `22` (8), `23` (9), `24` (10), `25` (11) - There are I think `250` families. - So, a `CROSS JOIN` will give us `11 * 250 = 2750` rows. ```sql SELECT COUNT(*) FROM (WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates) as count; ``` 11 dates for each family. ``` sqlite> SELECT COUNT(*) FROM (WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ) SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates) as count; +----------+ | COUNT(*) | +----------+ | 2750 | +----------+ Run Time: real 0.001 user 0.000407 sys 0.000000 sqlite> ``` Now, we simply have to join the `deliveries_assigned` table with the above table. Why? Because we need to map which dates are assigned and which dates are missing. We need to do a which join? LEFT, RIGHT or INNER LEFT WHy? Because, the left table will have all the dates, and the right table will have the assigned dates. **We need all the records from the `left` table (the combination, cross join table)** We need all the rows in the `left` or the combination(cross join) of date and family table as to map which dates are assigned and which dates are missing. If the `left` table has a record, then it means the date is assigned. If the `right` table has a record i.e. the gift is assigned, then it means the date is not assigned. Hence, we can simply then filter out the relations with `NULL` as the `family_id` in the `deliveries_assigned` table. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date ``` ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date LIMIT 30; +-----------+---------------+------------+----+-----------+------------+-------------------------+ | family_id | family_name | gift_date | id | family_id | gift_date | gift_name | +-----------+---------------+------------+----+-----------+------------+-------------------------+ | 1 | Isla Martinez | 2025-12-15 | 12 | 1 | 2025-12-15 | fudge bites | | 1 | Isla Martinez | 2025-12-16 | 13 | 1 | 2025-12-16 | holiday sticker sheet | | 1 | Isla Martinez | 2025-12-17 | | | | | | 1 | Isla Martinez | 2025-12-18 | 14 | 1 | 2025-12-18 | hot cocoa bombs | | 1 | Isla Martinez | 2025-12-19 | 15 | 1 | 2025-12-19 | honey roasted nuts | | 1 | Isla Martinez | 2025-12-20 | 16 | 1 | 2025-12-20 | holiday mug | | 1 | Isla Martinez | 2025-12-21 | 17 | 1 | 2025-12-21 | white chocolate candies | | 1 | Isla Martinez | 2025-12-22 | 18 | 1 | 2025-12-22 | puzzle book | | 1 | Isla Martinez | 2025-12-23 | 19 | 1 | 2025-12-23 | snowman plush | | 1 | Isla Martinez | 2025-12-24 | 20 | 1 | 2025-12-24 | scented hand cream | | 1 | Isla Martinez | 2025-12-25 | 21 | 1 | 2025-12-25 | vanilla bean wafers | | 2 | Nolan Garcia | 2025-12-15 | 36 | 2 | 2025-12-15 | mini marshmallow tubes | | 2 | Nolan Garcia | 2025-12-16 | 37 | 2 | 2025-12-16 | white chocolate candies | | 2 | Nolan Garcia | 2025-12-17 | 38 | 2 | 2025-12-17 | gingerbread cookie kit | | 2 | Nolan Garcia | 2025-12-18 | 39 | 2 | 2025-12-18 | family card game | | 2 | Nolan Garcia | 2025-12-19 | | | | | | 2 | Nolan Garcia | 2025-12-20 | 40 | 2 | 2025-12-20 | santa hat | | 2 | Nolan Garcia | 2025-12-21 | 41 | 2 | 2025-12-21 | holiday sticker sheet | | 2 | Nolan Garcia | 2025-12-22 | | | | | | 2 | Nolan Garcia | 2025-12-23 | 42 | 2 | 2025-12-23 | pecan praline bites | | 2 | Nolan Garcia | 2025-12-24 | | | | | | 2 | Nolan Garcia | 2025-12-25 | 43 | 2 | 2025-12-25 | santa hat | | 3 | Yara Chen | 2025-12-15 | 57 | 3 | 2025-12-15 | peppermint bark bites | | 3 | Yara Chen | 2025-12-16 | | | | | | 3 | Yara Chen | 2025-12-17 | | | | | | 3 | Yara Chen | 2025-12-18 | 58 | 3 | 2025-12-18 | cheddar popcorn | | 3 | Yara Chen | 2025-12-19 | | | | | | 3 | Yara Chen | 2025-12-20 | 59 | 3 | 2025-12-20 | festive notepad | | 3 | Yara Chen | 2025-12-21 | 60 | 3 | 2025-12-21 | fruit assortment | | 3 | Yara Chen | 2025-12-22 | | | | | +-----------+---------------+------------+----+-----------+------------+-------------------------+ ... ... | 249 | Jude Bautista | 2025-12-24 | | | | | | 249 | Jude Bautista | 2025-12-25 | 5073 | 249 | 2025-12-25 | almond brittle | | 250 | Bianca Muller | 2025-12-15 | 5086 | 250 | 2025-12-15 | cocoa mix bundle | | 250 | Bianca Muller | 2025-12-16 | 5087 | 250 | 2025-12-16 | cookie decorating kit | | 250 | Bianca Muller | 2025-12-17 | 5088 | 250 | 2025-12-17 | shortbread cookie tin | | 250 | Bianca Muller | 2025-12-18 | | | | | | 250 | Bianca Muller | 2025-12-19 | 5089 | 250 | 2025-12-19 | snowflake candle | | 250 | Bianca Muller | 2025-12-20 | | | | | | 250 | Bianca Muller | 2025-12-21 | 5090 | 250 | 2025-12-21 | trail mix trio | | 250 | Bianca Muller | 2025-12-22 | 5091 | 250 | 2025-12-22 | shortbread cookie tin | | 250 | Bianca Muller | 2025-12-23 | | | | | | 250 | Bianca Muller | 2025-12-24 | | | | | | 250 | Bianca Muller | 2025-12-25 | 5092 | 250 | 2025-12-25 | gingerbread cookie kit | +-----------+-------------------+------------+------+-----------+------------+----------------------------+ Run Time: real 0.028 user 0.013392 sys 0.013096 sqlite> ``` This will give all the gifts that have been assigned as well as not assigned for each family on each date from `2025-12-15` to `2025-12-25`. Now, you can see, what we need and what we don't we simply can get the gifts or records with the `family_id` in the `deliveries_assigned` table as `NULL`, since there was no record for the family_id in the `deliveries_assigned` table for that date. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ``` So, a simple `WHERE` clause with deliveries_assigned.family_id `IS NULL` will give us the missing dates for each family. ``` sqlite> WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT * FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ...> ; +-----------+-------------------+------------+----+-----------+-----------+-----------+ | family_id | family_name | gift_date | id | family_id | gift_date | gift_name | +-----------+-------------------+------------+----+-----------+-----------+-----------+ | 1 | Isla Martinez | 2025-12-17 | | | | | | 2 | Nolan Garcia | 2025-12-19 | | | | | | 2 | Nolan Garcia | 2025-12-22 | | | | | | 2 | Nolan Garcia | 2025-12-24 | | | | | | 3 | Yara Chen | 2025-12-16 | | | | | | 3 | Yara Chen | 2025-12-17 | | | | | | 3 | Yara Chen | 2025-12-19 | | | | | | 3 | Yara Chen | 2025-12-22 | | | | | | 4 | Tariq Nguyen | 2025-12-16 | | | | | ... ... | 247 | Malik Kim | 2025-12-18 | | | | | | 247 | Malik Kim | 2025-12-20 | | | | | | 247 | Malik Kim | 2025-12-21 | | | | | | 248 | Tariq Flores | 2025-12-16 | | | | | | 248 | Tariq Flores | 2025-12-17 | | | | | | 248 | Tariq Flores | 2025-12-18 | | | | | | 249 | Jude Bautista | 2025-12-16 | | | | | | 249 | Jude Bautista | 2025-12-20 | | | | | | 249 | Jude Bautista | 2025-12-21 | | | | | | 249 | Jude Bautista | 2025-12-23 | | | | | | 249 | Jude Bautista | 2025-12-24 | | | | | | 250 | Bianca Muller | 2025-12-18 | | | | | | 250 | Bianca Muller | 2025-12-20 | | | | | | 250 | Bianca Muller | 2025-12-23 | | | | | | 250 | Bianca Muller | 2025-12-24 | | | | | +-----------+-------------------+------------+----+-----------+-----------+-----------+ Run Time: real 0.014 user 0.011508 sys 0.002046 sqlite> ``` All right, we now need to order by `unassigned_date` and `name` which are the `dates` from the `combination` table and the `family_name` from the `families` table. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT combination.gift_date as unassigned_date, family_name FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ORDER BY unassigned_date, family_name; ``` ``` +-----------------+-------------------+ | unassigned_date | family_name | +-----------------+-------------------+ | 2025-12-15 | Adil Rossi | | 2025-12-15 | Aisha Connor | | 2025-12-15 | Amina Perez | | 2025-12-15 | Amina Wong | | 2025-12-15 | Andre Flores | | 2025-12-15 | Anya Singh | | 2025-12-15 | Arjun Wong | | 2025-12-15 | Bianca Connor | | 2025-12-15 | Caleb Petrov | | 2025-12-15 | Caleb Roberts | | 2025-12-15 | Carmen Carter | | 2025-12-15 | Carmen Garcia | | 2025-12-15 | Casey Flores | | 2025-12-15 | Chi Hughes | | 2025-12-15 | Clara Johnson | | 2025-12-15 | Dara Bautista | | 2025-12-15 | David Ramirez | | 2025-12-15 | Elias Petrov | | 2025-12-15 | Elias Petrov | | 2025-12-15 | Ethan Flores | | 2025-12-15 | Eva Gonzalez | ... ... | 2025-12-25 | Owen Park | | 2025-12-25 | Priya Khan | | 2025-12-25 | Rafael Singh | | 2025-12-25 | Ravi Abdallah | | 2025-12-25 | Ravi Mitchell | | 2025-12-25 | Rosa Turner | | 2025-12-25 | Sara Jensen | | 2025-12-25 | Sara Lopez | | 2025-12-25 | Sara Rossi | | 2025-12-25 | Sarah Phillips | | 2025-12-25 | Seth Garcia | | 2025-12-25 | Sienna Lopez | | 2025-12-25 | Sofia Nakamura | | 2025-12-25 | Tariq Nguyen | | 2025-12-25 | Uma Ali | | 2025-12-25 | Uma Phillips | | 2025-12-25 | Yara Chen | | 2025-12-25 | Yara Rossi | | 2025-12-25 | Yusuf Ali | | 2025-12-25 | Yusuf Hansen | | 2025-12-25 | Yusuf Perez | | 2025-12-25 | Yusuf Rossi | | 2025-12-25 | Zara Khan | +-----------------+-------------------+ Run Time: real 0.008 user 0.004613 sys 0.002845 sqlite> ``` Phew! Ok, that looks a mamoth query. ```sql WITH RECURSIVE dates(gift_date) AS ( SELECT '2025-12-15' UNION ALL SELECT date(gift_date, '+1 day') FROM dates WHERE gift_date < '2025-12-25' ), combination AS ( SELECT families.id AS family_id, families.family_name as family_name, dates.gift_date as gift_date FROM families CROSS JOIN dates ) SELECT combination.gift_date as unassigned_date, family_name FROM combination LEFT JOIN deliveries_assigned ON deliveries_assigned.family_id = combination.family_id AND deliveries_assigned.gift_date = combination.gift_date WHERE deliveries_assigned.family_id IS NULL ORDER BY unassigned_date, family_name; ``` So, to recap - Generate a table of dates from `2025-12-15` to `2025-12-25` using a recursive CTE. - Create a table of combinations of `families` and `dates` using a cross join. - Left join the `deliveries_assigned` table with the combination table. - Filter out the rows where the `family_id` in the `deliveries_assigned` table is `NULL`. - Order the results by `unassigned_date` and `name`, respectively, both in ascending order. Simple right? That's it from day 6. It getting serious out there! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-5 Advent of SQL Day 5 - EchoTrack Wrapped It is day 5 of advent of SQL. Let's get rollin. It looks like a good problem. I am excited! Here's the SQL to get starte Sat, 20 Dec 2025 00:00:00 UTC ## Advent of SQL Day 5 - EchoTrack Wrapped It is day 5 of advent of SQL. Let's get rollin. It looks like a good problem. I am excited! Here's the SQL to get started. ```sql DROP TABLE IF EXISTS listening_logs; CREATE TABLE listening_logs ( id INTEGER PRIMARY KEY, user_name TEXT, artist TEXT, played_at TIMESTAMP, content_type TEXT ); INSERT INTO listening_logs (id, user_name, artist, played_at, content_type) VALUES (1, 'Zoe Garcia', 'Arijit Singh', '2025-04-08 00:21:53', 'song'), (2, 'Zoe Garcia', 'Huberman Lab', '2025-11-10 19:18:47', 'podcast'), (3, 'Zoe Garcia', 'Huberman Lab', '2025-01-20 15:31:02', 'podcast'), (4, 'Zoe Garcia', 'Arijit Singh', '2025-01-06 17:33:11', 'song'), (5, 'Zoe Garcia', 'Candace', '2025-03-06 14:07:54', 'podcast'), (6, 'Zoe Garcia', 'Arijit Singh', '2025-06-05 17:57:59', 'song'), (7, 'Zoe Garcia', 'Huberman Lab', '2025-01-01 20:05:22', 'podcast'), (8, 'Zoe Garcia', 'Huberman Lab', '2025-11-01 12:04:03', 'podcast'), (9, 'Zoe Garcia', 'Arijit Singh', '2025-09-28 12:42:12', 'song'), (10, 'Zoe Garcia', 'The Ben Shapiro Show', '2025-09-15 01:05:15', 'podcast'), (11, 'Zoe Garcia', 'Arijit Singh', '2025-04-26 05:31:02', 'song'), (12, 'Zoe Garcia', 'Arijit Singh', '2025-10-13 17:34:03', 'song'), (13, 'Zoe Garcia', 'Mariah Carey', '2025-01-20 11:21:37', 'song'), (14, 'Zoe Garcia', 'Arijit Singh', '2025-11-28 03:55:31', 'song'), (15, 'Zoe Garcia', 'Arijit Singh', '2025-07-17 05:18:16', 'song'), (16, 'Zoe Garcia', 'Arijit Singh', '2025-08-20 02:07:45', 'song'), (17, 'Zoe Garcia', 'Kendrick Lamar', '2025-02-16 13:25:27', 'song'), (18, 'Zoe Garcia', 'Huberman Lab', '2025-08-13 19:55:00', 'podcast'), (19, 'Zoe Garcia', 'Bruno Mars', '2025-09-13 07:09:43', 'song'), (20, 'Zoe Garcia', 'Arijit Singh', '2025-04-12 06:30:44', 'song'); ``` Let's open a SQLite shell and get started. ``` $ sqlite3 SQLite version 3.50.4 2025-07-30 19:33:53 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day5-inserts.sql sqlite> .schema CREATE TABLE listening_logs ( id INTEGER PRIMARY KEY, user_name TEXT, artist TEXT, played_at TIMESTAMP, content_type TEXT ); sqlite> .mode table sqlite> SELECT * FROM listening_logs LIMIT 20; +----+------------+----------------------+---------------------+--------------+ | id | user_name | artist | played_at | content_type | +----+------------+----------------------+---------------------+--------------+ | 1 | Zoe Garcia | Arijit Singh | 2025-04-08 00:21:53 | song | | 2 | Zoe Garcia | Huberman Lab | 2025-11-10 19:18:47 | podcast | | 3 | Zoe Garcia | Huberman Lab | 2025-01-20 15:31:02 | podcast | | 4 | Zoe Garcia | Arijit Singh | 2025-01-06 17:33:11 | song | | 5 | Zoe Garcia | Candace | 2025-03-06 14:07:54 | podcast | | 6 | Zoe Garcia | Arijit Singh | 2025-06-05 17:57:59 | song | | 7 | Zoe Garcia | Huberman Lab | 2025-01-01 20:05:22 | podcast | | 8 | Zoe Garcia | Huberman Lab | 2025-11-01 12:04:03 | podcast | | 9 | Zoe Garcia | Arijit Singh | 2025-09-28 12:42:12 | song | | 10 | Zoe Garcia | The Ben Shapiro Show | 2025-09-15 01:05:15 | podcast | | 11 | Zoe Garcia | Arijit Singh | 2025-04-26 05:31:02 | song | | 12 | Zoe Garcia | Arijit Singh | 2025-10-13 17:34:03 | song | | 13 | Zoe Garcia | Mariah Carey | 2025-01-20 11:21:37 | song | | 14 | Zoe Garcia | Arijit Singh | 2025-11-28 03:55:31 | song | | 15 | Zoe Garcia | Arijit Singh | 2025-07-17 05:18:16 | song | | 16 | Zoe Garcia | Arijit Singh | 2025-08-20 02:07:45 | song | | 17 | Zoe Garcia | Kendrick Lamar | 2025-02-16 13:25:27 | song | | 18 | Zoe Garcia | Huberman Lab | 2025-08-13 19:55:00 | podcast | | 19 | Zoe Garcia | Bruno Mars | 2025-09-13 07:09:43 | song | | 20 | Zoe Garcia | Arijit Singh | 2025-04-12 06:30:44 | song | +----+------------+----------------------+---------------------+--------------+ sqlite> sqlite> SELECT COUNT(*) FROM listening_logs; +----------+ | COUNT(*) | +----------+ | 18174 | +----------+ sqlite> ``` OK! We have around 18k records in a single table! That's a lot but not not much! Let's see what we have to do ## Problem > Write a query that returns the top 3 artists per user. Order the results by the most played Alright, this is quite a problem to solve, if you are thinking, it easy peasy, then hold on! We clearly see, that we have 2 columns of our interest. 1. `user_name` 2. `artist` We need to group for each user his most played artists, and we need to rank those top 3 artist per user. And each entry is a song or podcast that the user has litened to. We need to aggregate, group, and then rank, and then what? How would you chunk out the top three? It's time to put your SQL glasses and gloves on, it's getting colder! ### Counting Artists Per User Let's take one step at a time, we need to first count how many times each artist has been played for that user. ```sql SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` OK, simple right? Select all the usernames, artist and then group by the username and the artist and count the number of times the user had played that artist. Simply then order by username, the playcount and the artist (if there is a tie in count, I think we can choose the artist name as the breaker) But it gives all the artists, not just the top 3, it ranks them in the decreasing order of the number of plays, but we only want to list the top 3 per user. That's tricky! ### With SELF JOIN What if we can join the table with itself? Then we can compare the number of times the user has played the artist, with the number of times the user has played another artist. Then we can remove the artist if it has played less than 3 times, this then will filter out the top 3 for us. ```sql SELECT a.user_name, a.artist AS current_artist, a.play_count AS current_plays, b.artist AS other_artist, b.play_count AS other_plays FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name ORDER BY a.user_name, a.play_count DESC, b.play_count DESC; ``` This would create a cross join of sorts between the same table. - Select the required columns (user_name, artist and count) Look at this part This is `a` ```sql -- a SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist ``` Then we need to join this with itself - Select the required columns from the same table (user_name, artist and count) This is `b` ```sql -- b SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist ``` Both `a` and `b` are the same, just that we want a cross join of sorts. And then we need to join `a` and `b` - Join with itself on the user_name - Order by user_name, play_count desc, artist ```sql SELECT a.user_name, a.artist AS current_artist, a.play_count AS current_plays, b.artist AS other_artist, b.play_count AS other_plays FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name ORDER BY a.user_name, a.play_count DESC, b.play_count DESC; ``` Now, we want to keep the rows where the `b` table has play count greater than `a` table, or if they are equal, then if the `b` table has artist less than `a` table. To do that we can continue the `JOIN` condition with `AND` and add `b.play_count > a.play_count` and `b.artist < a.artist` in case of a tie. The idea here is subtle: - For a given artist `a`, we count how many artists `b` (for the same user) have more plays, or the same plays but come earlier alphabetically. - If fewer than 3 artists beat `a`, then `a` must be in the top 3. So the query becomes this: ```sql SELECT a.user_name, a.artist, a.play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name AND ( b.play_count > a.play_count OR (b.play_count = a.play_count AND b.artist < a.artist) ) GROUP BY a.user_name, a.artist, a.play_count HAVING COUNT(b.artist) < 3 ORDER BY a.user_name, a.play_count DESC, a.artist; ``` ``` sqlite> SELECT a.user_name, a.artist, a.play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) a LEFT JOIN ( SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ) b ON a.user_name = b.user_name AND ( b.play_count > a.play_count OR (b.play_count = a.play_count AND b.artist < a.artist) ) GROUP BY a.user_name, a.artist, a.play_count HAVING COUNT(b.artist) < 3 ORDER BY a.user_name, a.play_count DESC, a.artist; +-------------------+--------------------------------------------+------------+ | user_name | artist | play_count | +-------------------+--------------------------------------------+------------+ | Abigail Hernandez | Ed Sheeran | 78 | | Abigail Hernandez | Rotten Mango | 15 | | Abigail Hernandez | Billie Eilish | 4 | | Adrian Cox | Kendrick Lamar | 128 | | Adrian Cox | Stuff You Should Know | 30 | | Adrian Cox | Fuerza Regida | 6 | | Alex Rivera | Ed Sheeran | 274 | | Alex Rivera | Call Her Daddy (Alex Cooper) | 42 | | Alex Rivera | Green Day | 11 | | Anders Nilsson | Snow Patrol | 101 | | Anders Nilsson | SmartLess | 29 | | Anders Nilsson | Blink-182 | 5 | | Anthony King | Pentatonix | 114 | | Anthony King | The Tucker Carlson Show | 14 | | Anthony King | Angels & Airwaves | 5 | ... ... | Zara Sheikh | Green Day | 138 | | Zara Sheikh | This Past Weekend w Theo Von | 20 | | Zara Sheikh | The Beatles | 7 | | Zoe Garcia | Arijit Singh | 50 | | Zoe Garcia | Huberman Lab | 14 | | Zoe Garcia | Kendrick Lamar | 5 | | Zoe Wilson | Pentatonix | 65 | | Zoe Wilson | The Mel Robbins Podcast | 14 | | Zoe Wilson | Angels & Airwaves | 4 | | Zuri Okafor | Kendrick Lamar | 96 | | Zuri Okafor | The Tim Dillon Show | 14 | | Zuri Okafor | Ed Sheeran | 5 | +-------------------+--------------------------------------------+------------+ Run Time: real 0.088 user 0.082510 sys 0.003999 ``` This is the final query, it looks long, it might be not the best way to do it, but its definitely not the worst way to do it. ### With Window Functions Ok! I don't know window functions, but I searched and found that we could partition things before we group them or order them in the final result set. That's what we want right? We had grouped the logs for each username and artist and counted the number of plays. Now we want to rank the artists for each user in the decreasing order of number of plays. So, we start with the same thing: ```sql SELECT user_name, artist, COUNT(*) AS play_count FROM listening_logs GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` You can see that we have `user_name` column, what if we could separate out the users, and then we could rank the artists for each user separately. For that we can use `ROW_NUMBER` window function. This function needs a `PARTITION BY` clause which lets us create separate partitions based on certain columns, and then we can rank them using the `ORDER BY` clause as we do with ordinary statement. That becomes the row_number, which we can use as a rank to rank each artist for a user based on the number of time the user has played him/her. ```sql SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ``` Here, `ROW_NUMBER` is a window function that assigns a rank to each row in the result set. It partitions the result set by `user_name` and orders the rows in the decreasing order of `COUNT(*)` and `artist` name. So, imagine this table: ``` +-------------------+-------------------------------------+------------+-------+ | user_name | artist | play_count | ranks | +-------------------+-------------------------------------+------------+-------+ | Abigail Hernandez | Ed Sheeran | 78 | 1 | | Abigail Hernandez | Rotten Mango | 15 | 2 | | Abigail Hernandez | Billie Eilish | 4 | 3 | | Abigail Hernandez | Hans Zimmer | 4 | 4 | | Abigail Hernandez | John Legend | 3 | 5 | | Abigail Hernandez | John Williams | 3 | 6 | | Abigail Hernandez | The Beatles | 3 | 7 | | Abigail Hernandez | The Rolling Stones | 3 | 8 | | Abigail Hernandez | Angels & Airwaves | 2 | 9 | | Abigail Hernandez | Bad Bunny | 2 | 10 | | Abigail Hernandez | Beyonce | 2 | 11 | | Abigail Hernandez | Coldplay | 2 | 12 | | Abigail Hernandez | Foo Fighters | 2 | 13 | | Abigail Hernandez | Fuerza Regida | 2 | 14 | | Abigail Hernandez | Kendrick Lamar | 2 | 15 | | Abigail Hernandez | Ludovico Einaudi | 2 | 16 | | Abigail Hernandez | Mariah Carey | 2 | 17 | | Abigail Hernandez | Pentatonix | 2 | 18 | | Abigail Hernandez | SmartLess | 2 | 19 | | Abigail Hernandez | The Weeknd | 2 | 20 | | Abigail Hernandez | Adele | 1 | 21 | | Abigail Hernandez | Ariana Grande | 1 | 22 | | Abigail Hernandez | Armchair Expert With Dax Shepard | 1 | 23 | | Abigail Hernandez | Bruno Mars | 1 | 24 | | Abigail Hernandez | Candace | 1 | 25 | | Abigail Hernandez | Crime, Conspiracy, Cults and Murder | 1 | 26 | | Abigail Hernandez | Green Day | 1 | 27 | | Abigail Hernandez | Matt and Shanes Secret Podcast | 1 | 28 | | Abigail Hernandez | On Purpose With Jay Shetty | 1 | 29 | | Abigail Hernandez | Snow Patrol | 1 | 30 | | Abigail Hernandez | Sufjan Stevens | 1 | 31 | | Abigail Hernandez | Taylor Swift | 1 | 32 | | Abigail Hernandez | The Mel Robbins Podcast | 1 | 33 | | Abigail Hernandez | Unseen | 1 | 34 | | Adrian Cox | Kendrick Lamar | 128 | 1 | | Adrian Cox | Stuff You Should Know | 30 | 2 | | Adrian Cox | Fuerza Regida | 6 | 3 | | Adrian Cox | Pentatonix | 6 | 4 | | Adrian Cox | Taylor Swift | 6 | 5 | | Adrian Cox | Snow Patrol | 5 | 6 | +-------------------+-------------------------------------+------------+-------+ ``` Now, we have ranked the artist for each user, there are `34` artist played by `Abigail Hernandez` so there are `34` ranks. Now, we need to filter out the top 3 artist for each user. That would be simple right? Just add the `WHERE` clause: ```sql SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs WHERE ranks <= 3 GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; ``` Well not really! ``` sqlite> SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) FROM listening_logs WHERE ranks <= 3 GROUP BY user_name, artist ORDER BY user_name, play_count DESC, artist; Run Time: real 0.000 user 0.000110 sys 0.000000 Parse error: misuse of aliased window function ranks ``` We can't filter the window function column with a `WHERE` clause. At the time the `WHERE` clause is evaluated, the `SELECT` list (including ranks) has not been computed yet. So ranks doesn't exist yet! So, now what, so close yet so far! Let's wrap the `SELECT` in a subquery: ```sql SELECT user_name, artist, play_count FROM ( SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ) ranked WHERE ranks <= 3 ORDER BY user_name, play_count DESC, artist; ``` And there we have it! We wrapped the `SELECT` in a subquery, and now we can filter the window function column with a `WHERE` clause. We can even do it with a `CTE` i.e. common table expression. That's just a subquery with a name. Not sure when it could be handy, but here it is. ```sql WITH ranked AS ( SELECT user_name, artist, COUNT(*) AS play_count, ROW_NUMBER() OVER ( PARTITION BY user_name ORDER BY COUNT(*) DESC, artist ) AS ranks FROM listening_logs GROUP BY user_name, artist ) SELECT user_name, artist, play_count FROM ranked WHERE ranks <= 3 ORDER BY user_name, play_count DESC, artist; ``` So, that is day 5, ok that is getting a little tricky now! Off to day 6. sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-4 Advent of SQL Day 4 WinterFest Volunteers It is day 4 of advent of SQL. No fuss, straight to the problem, the elves and humans are getting dumber as the days pr Fri, 19 Dec 2025 00:00:00 UTC ## Advent of SQL Day 4 WinterFest Volunteers It is day 4 of advent of SQL. No fuss, straight to the problem, the elves and humans are getting dumber as the days progress. Let's download the SQL inserts for the day. And load it into a SQLite shell. ```sql DROP TABLE IF EXISTS official_shifts; DROP TABLE IF EXISTS last_minute_signups; CREATE TABLE official_shifts ( id INT PRIMARY KEY, volunteer_name TEXT, role TEXT, shift_time TEXT, age_group TEXT, code TEXT ); CREATE TABLE last_minute_signups ( id INT PRIMARY KEY, volunteer_name TEXT, assigned_task TEXT, time_slot TEXT ); INSERT INTO official_shifts (id, volunteer_name, role, shift_time, age_group, code) VALUES (1, 'Jude Thompson', 'choir_assistant', '12:00 PM', 'senior', NULL), (2, 'Mateo Cruz', 'choir_assistant', '12:00 PM', 'senior', NULL), (3, 'Olivia Dubois', 'choir_assistant', '2:00 PM', 'teen', 'A1'), (4, 'Jeff Bezos', 'choir_assistant', '10:00 AM', 'adult', 'X7'), (5, 'Kian Rahimi', 'stage_setup', '12:00 PM', 'adult', 'X7'), (6, 'Haruto Sato', 'cocoa_station', '10:00 AM', 'adult', 'X7'), (7, 'Uma Singh', 'parking_support', '10:00 AM', 'adult', NULL), (8, 'Owen Scott', 'parking_support', '10:00 AM', 'adult', 'X7'), (9, 'Adil Rahman', 'stage_setup', '2:00 PM', 'adult', 'A1'), (10, 'Aaron Diaz', 'choir_assistant', '2:00 PM', 'senior', 'X7'), (11, 'Carter Lewis', 'cocoa_station', '10:00 AM', 'senior', 'B2'), (12, 'Anya Pavlov', 'stage_setup', '10:00 AM', 'senior', 'OLD'), (13, 'Ethan Brown', 'stage_setup', '2:00 PM', 'adult', 'A1'), (14, 'Lucia Fernandez', 'choir_assistant', '12:00 PM', 'senior', 'X7'), (15, 'Casey Morgan', 'choir_assistant', '12:00 PM', 'teen', 'OLD'); INSERT INTO last_minute_signups (id, volunteer_name, assigned_task, time_slot) VALUES (1, 'Jude Thompson', 'Choir', 'noon'), (2, 'Mateo Cruz', 'choir', 'noon'), (3, 'Olivia Dubois', 'choir', '2 PM'), (4, 'Jeff Bezos', 'choir assistant', '10AM'), (5, 'Kian Rahimi', 'stage setup', 'noon'), (6, 'Haruto Sato', 'cocoa station', '10AM'), (7, 'Uma Singh', 'parking_support', '10AM'), (8, 'Owen Scott', 'parking', '10AM'), (9, 'Adil Rahman', 'Stage-Setup', '2 PM'), (10, 'Aaron Diaz', 'Choir', '2 PM'), (11, 'Carter Lewis', 'Cocoa Station', '10AM'), (12, 'Anya Pavlov', 'stage_setup', '10AM'), (13, 'Olivia Brown', 'stage setup', '2 PM'), (14, 'Lena Fischer', 'cocoa station', '2 pm'), (15, 'Nolan Murphy', 'parking-support', '10AM'); ``` Once the data is loaded, let's sneak peak. ```sql SELECT * FROM official_shifts LIMIT 15; ``` ```sql SELECT * FROM last_minute_signups LIMIT 15; ``` Let's count how many rows in each table we have: ```sql SELECT COUNT(*) FROM official_shifts LIMIT 15; SELECT COUNT(*) FROM last_minute_signups LIMIT 15; ``` Alright, the data is visible and we can head on to the problem statement. ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day4_inserts.sql sqlite> .schema CREATE TABLE official_shifts ( id INT PRIMARY KEY, volunteer_name TEXT, role TEXT, shift_time TEXT, age_group TEXT, code TEXT ); CREATE TABLE last_minute_signups ( id INT PRIMARY KEY, volunteer_name TEXT, assigned_task TEXT, time_slot TEXT ); sqlite> SELECT * FROM official_shifts LIMIT 15; 1|Jude Thompson|choir_assistant|12:00 PM|senior| 2|Mateo Cruz|choir_assistant|12:00 PM|senior| 3|Olivia Dubois|choir_assistant|2:00 PM|teen|A1 4|Jeff Bezos|choir_assistant|10:00 AM|adult|X7 5|Kian Rahimi|stage_setup|12:00 PM|adult|X7 6|Haruto Sato|cocoa_station|10:00 AM|adult|X7 7|Uma Singh|parking_support|10:00 AM|adult| 8|Owen Scott|parking_support|10:00 AM|adult|X7 9|Adil Rahman|stage_setup|2:00 PM|adult|A1 10|Aaron Diaz|choir_assistant|2:00 PM|senior|X7 11|Carter Lewis|cocoa_station|10:00 AM|senior|B2 12|Anya Pavlov|stage_setup|10:00 AM|senior|OLD 13|Ethan Brown|stage_setup|2:00 PM|adult|A1 14|Lucia Fernandez|choir_assistant|12:00 PM|senior|X7 15|Casey Morgan|choir_assistant|12:00 PM|teen|OLD sqlite> .mode table sqlite> SELECT * FROM official_shifts LIMIT 15; +----+-----------------+-----------------+------------+-----------+------+ | id | volunteer_name | role | shift_time | age_group | code | +----+-----------------+-----------------+------------+-----------+------+ | 1 | Jude Thompson | choir_assistant | 12:00 PM | senior | | | 2 | Mateo Cruz | choir_assistant | 12:00 PM | senior | | | 3 | Olivia Dubois | choir_assistant | 2:00 PM | teen | A1 | | 4 | Jeff Bezos | choir_assistant | 10:00 AM | adult | X7 | | 5 | Kian Rahimi | stage_setup | 12:00 PM | adult | X7 | | 6 | Haruto Sato | cocoa_station | 10:00 AM | adult | X7 | | 7 | Uma Singh | parking_support | 10:00 AM | adult | | | 8 | Owen Scott | parking_support | 10:00 AM | adult | X7 | | 9 | Adil Rahman | stage_setup | 2:00 PM | adult | A1 | | 10 | Aaron Diaz | choir_assistant | 2:00 PM | senior | X7 | | 11 | Carter Lewis | cocoa_station | 10:00 AM | senior | B2 | | 12 | Anya Pavlov | stage_setup | 10:00 AM | senior | OLD | | 13 | Ethan Brown | stage_setup | 2:00 PM | adult | A1 | | 14 | Lucia Fernandez | choir_assistant | 12:00 PM | senior | X7 | | 15 | Casey Morgan | choir_assistant | 12:00 PM | teen | OLD | +----+-----------------+-----------------+------------+-----------+------+ sqlite> SELECT * FROM last_minute_signups LIMIT 15; +----+----------------+-----------------+-----------+ | id | volunteer_name | assigned_task | time_slot | +----+----------------+-----------------+-----------+ | 1 | Jude Thompson | Choir | noon | | 2 | Mateo Cruz | choir | noon | | 3 | Olivia Dubois | choir | 2 PM | | 4 | Jeff Bezos | choir assistant | 10AM | | 5 | Kian Rahimi | stage setup | noon | | 6 | Haruto Sato | cocoa station | 10AM | | 7 | Uma Singh | parking_support | 10AM | | 8 | Owen Scott | parking | 10AM | | 9 | Adil Rahman | Stage-Setup | 2 PM | | 10 | Aaron Diaz | Choir | 2 PM | | 11 | Carter Lewis | Cocoa Station | 10AM | | 12 | Anya Pavlov | stage_setup | 10AM | | 13 | Olivia Brown | stage setup | 2 PM | | 14 | Lena Fischer | cocoa station | 2 pm | | 15 | Nolan Murphy | parking-support | 10AM | +----+----------------+-----------------+-----------+ sqlite> SELECT count(*) FROM official_shifts; +----------+ | count(*) | +----------+ | 250 | +----------+ sqlite> SELECT count(*) FROM last_minute_signups; +----------+ | count(*) | +----------+ | 126 | +----------+ sqlite> ``` ## Problem Here's the challenge for day 4 > Using the official_shifts and last_minute_signups tables, create a combined de-duplicated volunteer list. > > Ensure the list has standardized role labels of Stage Setup, Cocoa Station, Parking Support, Choir Assistant, Snow Shoveling, Handwarmer Handout. > > Make sure that the timeslot formats follow John's official shifts format. What we have here is a official shift table which could have been entered by the system. However the `last_minute_shift` is messy and has been added from a sheet, so we need to clean it up and combine those two tables data into a single de-duplicated list of volunteers. Let's see what we got ```sql SELECT * FROM last_minute_signups; ``` Ok, we have 126 records and the columns are: 1. `volunteer_name` which doesn't look bad, 2. `assigned_task` which looks wonky 3. `time_slot` is just wild, we have wired definition of times there. And let's look at the `official_shifts` ```sql SELECT * FROM offical_shifts; ``` This looks neat and tidy, nothing looking off from each other. So we need to make sure we are cleaning up the `last_minute_signups` before we merge them. ```sql SELECT DISTINCT assigned_task FROM last_minute_signups; ``` Ok, so casing is one thing we can see, `-` and ` ` space are the things to normalize, and then some inconsistent naming convention like `choir` and `choir assistant`, then `parking_support` and `parking`. We need to clean'em up. We can search for ```sql SELECT id, volunteer_name, time_slot, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task FROM last_minute_signups; ``` We just do a case match for: - `LIKE 'choir%'` which will match any case (`Choir`, `choir`) and also anything after `choir....` like `choir assistant` - `LIKE 'stage%'` which will match any case (`Stage`, `stage`) and also anything after `stage...` like `Stage-Setup`, `stage setup` or `stage_setup`. - `LIKE '%cocoa'` which will match any case (`Cocoa`, `cocoa`) and also anything before or after `...cocoa...` like `Cocoa Station`, `cocoa station`, etc. - `LIKE 'parking%'` which will match any case (`Parking`, `parking`) and also anything after `parking...` like `parking-support` or `parking_support`, etc. - `LIKE 'hand%'` which will match any case (`Hand`, `hand`) and also anything after `hand...` like `handwarmer handout`, `handwarmers`, `Handwarmer-Handout`, etc. - `LIKE '%shovel%'` which will match any case (`Shovel`, `shovel`) and also anything before and after `...shovel...` like `Snow-Shoveling`, `shovel`, `snow shoveling`, etc. Ok, now this looks unified for the `assigned_task`. ``` sqlite> SELECT id, volunteer_name, time_slot, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task FROM last_minute_signups; +-----+-------------------+-----------+-----------------+ | id | volunteer_name | time_slot | assigned_task | +-----+-------------------+-----------+-----------------+ | 1 | Jude Thompson | noon | choir_assistant | | 2 | Mateo Cruz | noon | choir_assistant | | 3 | Olivia Dubois | 2 PM | choir_assistant | | 4 | Jeff Bezos | 10AM | choir_assistant | | 5 | Kian Rahimi | noon | stage_setup | | 6 | Haruto Sato | 10AM | cocoa_station | | 7 | Uma Singh | 10AM | parking_support | | 8 | Owen Scott | 10AM | parking_support | | 9 | Adil Rahman | 2 PM | stage_setup | | 10 | Aaron Diaz | 2 PM | choir_assistant | | 11 | Carter Lewis | 10AM | cocoa_station | | 12 | Anya Pavlov | 10AM | stage_setup | | 13 | Olivia Brown | 2 PM | stage_setup | | 14 | Lena Fischer | 2 pm | cocoa_station | | 15 | Nolan Murphy | 10AM | parking_support | +-----+-------------------+-----------+-----------------+ ``` We need to make it for the time slot too. ```sql SELECT DISTINCT time_slot FROM last_minute_signups ``` ``` sqlite> SELECT DISTINCT time_slot FROM last_minute_signups; +-----------+ | time_slot | +-----------+ | noon | | 2 PM | | 10AM | | 2 pm | | 10 am | +-----------+ sqlite> ``` Since we have to follow the `official_shifts` let's check over there. ```sql SELECT distinct shift_time FROM official_shifts; ``` ``` sqlite> SELECT distinct shift_time FROM official_shifts; +------------+ | shift_time | +------------+ | 12:00 PM | | 2:00 PM | | 10:00 AM | +------------+ ``` Ok we have only 3 times to change. Alright, we have some small things to standardize. ```sql SELECT DISTINCT CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups; ``` ``` sqlite> SELECT DISTINCT CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' END AS time_slot FROM last_minute_signups; +-----------+ | time_slot | +-----------+ | 12:00 PM | | 2:00 PM | | 10:00 AM | +-----------+ sqlite> ``` So, we have simply standardize the time_slots. - `LIKE '2%'` will match any case but we need `LIKE` to match the `%` rest of the stuff after `2`. - `LIKE '10%'` will match any case but we need `LIKE` to match the `%` rest of the stuff after `10`. We can't keep it `LIKE '1%'` as it will match `1:00` as well - `LIKE 'noon'` will match any case of `noon` like `NOON` or `Noon`, etc. And we need to cast it to the `HH:MM AM or PM` format. So, now we can combine them. ```sql SELECT id, volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 PM' END AS shift_time FROM last_minute_signups; ``` Just changed the column names from `assigned_task` to `role` and `time_slot` to `shift_time` as per the name convention in the `official_shifts` table. Phew! its a long statement. ``` sqlite> SELECT id, volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups; +-----+-------------------+-----------------+-----------+ | id | volunteer_name | assigned_task | time_slot | +-----+-------------------+-----------------+-----------+ | 1 | Jude Thompson | choir_assistant | 12:00 PM | | 2 | Mateo Cruz | choir_assistant | 12:00 PM | | 3 | Olivia Dubois | choir_assistant | 2:00 PM | | 4 | Jeff Bezos | choir_assistant | 10:00 AM | | 5 | Kian Rahimi | stage_setup | 12:00 PM | | 6 | Haruto Sato | cocoa_station | 10:00 AM | | 7 | Uma Singh | parking_support | 10:00 AM | | 8 | Owen Scott | parking_support | 10:00 AM | | 9 | Adil Rahman | stage_setup | 2:00 PM | | 10 | Aaron Diaz | choir_assistant | 2:00 PM | | 11 | Carter Lewis | cocoa_station | 10:00 AM | | 12 | Anya Pavlov | stage_setup | 10:00 AM | | 13 | Olivia Brown | stage_setup | 2:00 PM | | 14 | Lena Fischer | cocoa_station | 2:00 PM | | 15 | Nolan Murphy | parking_support | 10:00 AM | +-----+-------------------+-----------------+-----------+ ``` SO, now we have the table of `last_minute_signups` cleaned up, just with select, we can update them if needed. We now need to combine the both tables, cleaned up `last_minute_signups` and the `official_shifts`, we can use `UNION` to take out the duplicates from the two selection. REMEMBER to order the rows correctly in both the tables. - volunteer_name - role - shift_time I don't think name should be same, but I am keeping it same for clarity. Why `UNION` - Because we have data in both the tables. - We don't have a relation in both of the tables, those are the same tables just that the columns are not cleaned or in proper format. - We want to grab all of them from one, all from other table, and remove the duplicates, that's a definition of `UNION` We can't use `UNION ALL` as it would include all the rows from both the tables without removing duplicates. ```sql SELECT volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups UNION SELECT volunteer_name, role, shift_time FROM official_shifts ORDER BY volunteer_name; ``` Ok, that is a mess, isn't it? ``` sqlite> SELECT volunteer_name, CASE WHEN assigned_task LIKE 'choir%' THEN 'choir_assistant' WHEN assigned_task LIKE 'stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE 'parking%' THEN 'parking_support' WHEN assigned_task LIKE 'hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS assigned_task, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS time_slot FROM last_minute_signups UNION SELECT volunteer_name, role, shift_time FROM official_shifts ORDER BY volunteer_name; +-------------------+-----------------+-----------+ | volunteer_name | assigned_task | time_slot | +-------------------+-----------------+-----------+ | Aaron Carter | parking_support | 2:00 PM | | Aaron Diaz | choir_assistant | 2:00 PM | | Aaron Diaz | choir_assistant | 2:00 PM | | Aaron Evans | cocoa_station | 2:00 PM | | Aaron Francis | hand_warmer | 2:00 PM | | Abigail Hernandez | stage_setup | 10:00 AM | | Adam King | stage_setup | 10:00 AM | | Adil Foster | stage_setup | 2:00 PM | | Adil Rahman | stage_setup | 2:00 PM | | Adil Rahman | stage_setup | 2:00 PM | | Adrian Cox | cocoa_station | 10:00 AM | | Aisha Bennett | cocoa_station | 12:00 PM | | Aisha Khan | choir_assistant | 12:00 PM | | Aisha Khan | choir_assistant | 12:00 PM | | Aisha Mohammed | cocoa_station | 2:00 PM | +-------------------+-----------------+-----------+ ``` There we have it. Let's count the number of distinct volunteers in the shifts. ```sql SELECT COUNT(*) FROM (SELECT volunteer_name, role, shift_time FROM official_shifts UNION SELECT volunteer_name, CASE WHEN assigned_task LIKE '%choir%' THEN 'choir_assistant' WHEN assigned_task LIKE '%stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE '%parking%' THEN 'parking_support' WHEN assigned_task LIKE '%hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS shift_time FROM last_minute_signups) AS volunteers; ``` Just counted the full union of the statement using `SELECT COUNT(*) FROM <THE FULL QUERY> AS volunteers) ``` sqlite> SELECT COUNT(*) FROM (SELECT volunteer_name, role, shift_time FROM official_shifts UNION SELECT volunteer_name, CASE WHEN assigned_task LIKE '%choir%' THEN 'choir_assistant' WHEN assigned_task LIKE '%stage%' THEN 'stage_setup' WHEN assigned_task LIKE '%cocoa%' THEN 'cocoa_station' WHEN assigned_task LIKE '%parking%' THEN 'parking_support' WHEN assigned_task LIKE '%hand%' THEN 'hand_warmer' WHEN assigned_task LIKE '%shovel%' THEN 'snow_showel' END AS role, CASE WHEN time_slot LIKE '2%' THEN '2:00 PM' WHEN time_slot LIKE 'noon' THEN '12:00 PM' WHEN time_slot LIKE '10%' THEN '10:00 AM' END AS shift_time FROM last_minute_signups) AS volunteers; +------------+ | volunteers | +------------+ | 284 | +------------+ sqlite> ``` So, we have around `284` rows. Looks good. Pinebrook can see the volunteer list now. The cleaned one. Off to day 5! sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-3 Advent of SQL Day 3 - Hotline Messages This is day 3 from the Advent of SQL Grab the SQL Statements Let's take the insert statements i.e. to create and populate Thu, 18 Dec 2025 00:00:00 UTC ## Advent of SQL Day 3 - Hotline Messages This is day 3 from the Advent of SQL ## Grab the SQL Statements Let's take the insert statements i.e. to create and populate tables and rows into the database. I am using SQLite. It works without any special shenanigans, as it was intended to used for Postgres, but the table and use case looks very simple, so nothing specific to Postgres used yet! We are good! Here is the SQL setup, if you want to play it in the playground here: ```sql DROP TABLE IF EXISTS hotline_messages; CREATE TABLE hotline_messages ( id INT PRIMARY KEY, caller_name TEXT, transcript TEXT, tag TEXT, status TEXT ); INSERT INTO hotline_messages (id, caller_name, transcript, tag, status) VALUES (1, 'Saanvi A.', 'I just found a refrigerator portal that leads to a disco party hosted by dancing llamas—please send help!', 'possible dragon', 'spam'), (2, 'Fatima Q.', 'Hi Santa, I would love a magical unicorn that lights up at night!', 'wish list', NULL), (3, 'Lillian Z.', 'Hi Santa, I would love the magical fairy garden set, please!', 'wish list', 'approved'), (4, 'Carter Y.', 'Thank you, Santa, for making Christmas so special with your wonderful spirit!', 'thank you', 'approved'), (5, 'Omar R.', 'Hi Santa, I would love a rainbow unicorn plushie that has a glittery horn!', 'wish list', 'approved'), (6, 'Diego Y.', 'Hi Santa, I would love a magical unicorn plushie that glows in the dark!', 'wish list', NULL), (7, 'Layla X.', 'Thank you, Santa, for spreading joy and magic every Christmas!', 'thank you', NULL), (8, 'Sophia K.', 'Santa, my cat said she wants to visit the candy cane forest next week.', NULL, NULL), (9, 'Eli H.', 'Hi Santa, I would love the magical fairy castle with twinkling lights!', 'wish list', 'approved'), (10, 'Logan F.', 'Santa, I think the reindeer are starting a band with the garden gnomes.', 'needs clarification', NULL), (11, 'Carlos P.', 'Thank you, Santa, for making Christmas so special every year!', 'thank you', NULL), (12, 'Zain G.', 'Thank you, Santa, for bringing joy to all the children around the world!', 'thank you', NULL), (13, 'Haruto R.', 'Thank you, Santa, for spreading so much joy and magic every Christmas!', 'thank you', 'approved'), (14, 'Oliver L.', 'Thank you, Santa, for spreading joy and making Christmas extra special!', 'thank you', NULL), (15, 'Luca M.', 'Hi Santa, could I please have the super cool glow-in-the-dark rocket ship?', 'wish list', NULL), (16, 'Samuel C.', 'sorry, Santa, my teddy bear said he wants to be a reindeer this year.', 'needs clarification', NULL); ``` Here's the setup I did to check the data. ```sql SELECT * FROM hotline_messages; ``` ```plaintext $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day3-inserts.sql sqlite> .schema CREATE TABLE hotline_messages ( id INT PRIMARY KEY, caller_name TEXT, transcript TEXT, tag TEXT, status TEXT ); sqlite> .mode table sqlite> SELECT * FROM hotline_messages LIMIT 10; +----+-------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 1 | Saanvi A. | I just found a refrigerator portal that leads to a disco par | possible dragon | spam | | | | ty hosted by dancing llamas—please send help! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 2 | Fatima Q. | Hi Santa, I would love a magical unicorn that lights up at n | wish list | | | | | ight! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 3 | Lillian Z. | Hi Santa, I would love the magical fairy garden set, please! | wish list | approved | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 4 | Carter Y. | Thank you, Santa, for making Christmas so special with your | thank you | approved | | | | wonderful spirit! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 5 | Omar R. | Hi Santa, I would love a rainbow unicorn plushie that has a | wish list | approved | | | | glittery horn! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 6 | Diego Y. | Hi Santa, I would love a magical unicorn plushie that glows | wish list | | | | | in the dark! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 7 | Layla X. | Thank you, Santa, for spreading joy and magic every Christma | thank you | | | | | s! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 8 | Sophia K. | Santa, my cat said she wants to visit the candy cane forest | | | | | | next week. | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 9 | Eli H. | Hi Santa, I would love the magical fairy castle with twinkli | wish list | approved | | | | ng lights! | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ | 10 | Logan F. | Santa, I think the reindeer are starting a band with the gar | needs clarification | | | | | den gnomes. | | | +----+-------------+--------------------------------------------------------------+---------------------+----------+ sqlite> ``` So, we just have one table, called `hotline_messages` and it has a few columns like 1. `caller_name` 2. `transcript` 3. `tag` 4. `status` What do we want to do with those? Well! Let's get into the problem statement. ## Problem Here goes the challenge for day 3 > Using the `hotline_messages` table, update any record that has "sorry" (case insensitive) in the transcript and doesn't currently have a status assigned to have a status of "approved". > > Then delete any records where the tag is "penguin prank", "time-loop advisory", "possible dragon", or "nonsense alert" or if the caller's name is "Test Caller". > > After updating and deleting the records as described, write a final query that returns how many messages currently have a status of "approved" and how many still need to be reviewed (i.e., status is NULL). It's divided into 3 parts, so we need three queries? Maybe, I don't want to have a single long query for doing all of these. And after reading it, it seems it should not be a single query, it can be 2 queries, one is for updation and the other for selection after those updates. So, we have to do three things. 1. Find the records which have `sorry` in the transcript text and mark their `status` as `approved` (What a lovely gesture) 2. Find all records with the tags as either `penguin prank`, `time-loop advisory`, `possible dragon`, or `nonsense alert` and even if the `caller_name` is `Test Caller` then delete those records, yes take'em out of my way. 3. After doing those 2 things, we have to count the number of records with `status` as `approved` and the number of records that are still not `approved` (they are in review or the status is `NULL` ) So, let's do them step by step. ### Be generous Let's be generous like Santa says and mark the records with `status` as `approved` whose transcript have the word `sorry` in them. Let those children be gifted their reward of being generous and humble. How do that in SQL, well let's first look at what we are updating. ```sql SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; ``` So, will it be sufficient? I think so. Because * `LIKE` is **case insensitive**, so it can catch `sorry`, `Sorry`, `SoRRY`, `sorrY` * `%` before and after will catch the word `sorry` in middle of the sentence and not necessarily in the start. I can see 104 rows selected with this condition. I always try to check before updation or deletion how many rows are affected. Because, sometimes we start `UPDATE hotline_messages SET status = 'approved'` and forget the where! This gets worse for delete believe me! ```sql SELECT count(*) FROM hotline_messages WHERE transcript LIKE '%sorry%'; SELECT count(*) FROM hotline_messages; ``` ```plaintext sqlite> SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; +------+--------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 16 | Samuel C. | sorry, Santa, my teddy bear said he wants to be a reindeer t | needs clarification | | | | | his year. | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 33 | Jacob F. | sorry, Hi Santa, I would love the magical unicorn plushie th | wish list | | | | | at glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 46 | Jun Y. | sorry, Hi Santa, I would love a magical unicorn stuffed anim | wish list | | | | | al that glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ sqlite> SELECT count(*) FROM hotline_messages WHERE transcript LIKE '%sorry%'; +----------+ | count(*) | +----------+ | 104 | +----------+ sqlite> SELECT count(*) FROM hotline_messages ;--WHERE transcript LIKE '%sorry%'; +----------+ | count(*) | +----------+ | 1067 | +----------+ sqlite> ``` So, once I know `104` rows will be affected out of `1067` I can create the update statement. We want to update the status and set it to `approved` for the rows which we selected just now (have `sorry` in the transcript text) ```sql UPDATE hotline_messages SET status = 'approved' WHERE transcript LIKE '%sorry%'; ``` Now, when we select again ```sql SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; ``` All approved! ```plaintext sqlite> SELECT * FROM hotline_messages WHERE transcript LIKE '%sorry%'; +------+--------------+--------------------------------------------------------------+---------------------+----------+ | id | caller_name | transcript | tag | status | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 16 | Samuel C. | sorry, Santa, my teddy bear said he wants to be a reindeer t | needs clarification | approved | | | | his year. | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 33 | Jacob F. | sorry, Hi Santa, I would love the magical unicorn plushie th | wish list | approved | | | | at glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ | 46 | Jun Y. | sorry, Hi Santa, I would love a magical unicorn stuffed anim | wish list | approved | | | | al that glows in the dark! | | | +------+--------------+--------------------------------------------------------------+---------------------+----------+ ``` To the next step then ### Remove Spam To reiterate the second part of the challenge > Then delete any records where the tag is "penguin prank", "time-loop advisory", "possible dragon", or "nonsense alert" or if the caller's name is "Test Caller". We basically need to Find all records with the tags as either `penguin prank`, `time-loop advisory`, `possible dragon`, or `nonsense alert` and even if the `caller_name` is `Test Caller` then delete those records, yes take'em out of my way. So, again, select first update or delete later. ```sql SELECT * FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); ``` Here `IN` is a great helper as we can do equivalent of this into a compact statement. ```sql SELECT * FROM hotline_messages WHERE tag = 'penguin prank' OR tag = 'time-loop advisory' OR tag = 'possible dragon' OR tag = 'nonsense alert'; ``` That is a lot of `OR tag =` that is saved by `IN` a list of values. Handy little operator. The count here is `68` ```sql SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); ``` ```plaintext sqlite> SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert'); +----------+ | count(*) | +----------+ | 68 | +----------+ sqlite> ``` Also we need to check if the `caller_name` is `Test Caller` It could be `OR` here ```sql SELECT * FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` That's it let's count the number of rows we will be deleting soon. ```plaintext SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` So, we have `89` rows to deleted after considering the spamy tags and test callers out. ```plaintext sqlite> SELECT count(*) FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; +----------+ | count(*) | +----------+ | 89 | +----------+ sqlite> ``` Let's get the spam outta here! ```sql DELETE FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; ``` Phew! Done 89 spammy records removed! Santa might be relieved. ```plaintext sqlite> DELETE FROM hotline_messages WHERE tag IN ('penguin prank', 'time-loop advisory', 'possible dragon', 'nonsense alert') OR caller_name = 'Test Caller'; sqlite> SELECT changes(); +-----------+ | changes() | +-----------+ | 89 | +-----------+ ``` The changes are done, now we simply have to select and count the things which are approved and in review. ### Count'em down * After doing those 2 things, we have to count the number of records with `status` as `approved` and the number of records that are still not `approved` (they are in review or the status is `NULL` ) So, we need to get the count of 1. Records with `status` as `approved` 2. Records with `status` as `NULL` #### Separate Queries This looks straight forward, you can write two separate queries for doing the things. ```sql SELECT COUNT(*) as approved_count FROM hotline_messages WHERE status = 'approved'; SELECT COUNT(*) as in_review_count FROM hotline_messages WHERE status IS NULL; ``` ```plaintext sqlite> SELECT COUNT(*) as approved_count FROM hotline_messages WHERE status = 'approved'; SELECT COUNT(*) as in_review_count FROM hotline_messages WHERE status IS NULL; +----------------+ | approved_count | +----------------+ | 477 | +----------------+ +-----------------+ | in_review_count | +-----------------+ | 501 | +-----------------+ sqlite> ``` #### Group by Status But however, can we do it in 1 query? Think! There are just 2 types of status right? Let's check ```sql SELECT DISTINCT status FROM hotline_messages; ``` Hmm! 2? `NULL` and `approved` ! ```sql sqlite> SELECT DISTINCT status FROM hotline_messages; +----------+ | status | +----------+ | | | approved | +----------+ ``` So we can simply do the same thing, but just group by `status` right? Like so: ```sql SELECT status, COUNT(*) as count FROM hotline_messages GROUP BY status; ``` And this should give us back the two rows with the count of `NULL` and `approved` . ```plaintext sqlite> SELECT status, COUNT(*) as count FROM hotline_messages GROUP BY status; +----------+-------+ | status | count | +----------+-------+ | | 501 | | approved | 477 | +----------+-------+ ``` Is there a better way? This looks a little wired! Like status is empty (NULL) and it kind of makes a little wired view for people to look at, can we do something different? #### Cases when then else This is simple use case for a `CASE WHEN ... THEN ... ELSE ... END` For each when we can check certain conditions and do certain things or do other thing. In this case, if the status is `approved`, we can increment the count to 1 or we can keep 0, similarly the other when can be used for grouping the count of status being `NULL` . ```sql SELECT COUNT(CASE WHEN status = 'approved' THEN 1 END) AS approved_count, COUNT(CASE WHEN status IS NULL THEN 1 END) AS in_review_count FROM hotline_messages; ``` What this will do is, for each row, we will count up the number of either `approved_count`or `in_review_count` depending on the value of the `status` cell. If that is `approved` we increment the `approved` count else if that is `NULL` we increment the `in_review_count`. Slick! ```sql sqlite> SELECT COUNT(CASE WHEN status = 'approved' THEN 1 END) AS approved_count, COUNT(CASE WHEN status IS NULL THEN 1 END) AS in_review_count FROM hotline_messages; +----------------+-----------------+ | approved_count | in_review_count | +----------------+-----------------+ | 477 | 501 | +----------------+-----------------+ sqlite> ``` That's it from day 3 hopefully Santa is happy, and in sight of getting madder as the elves get dumber. sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-2 SQLog: Advent of SQL Day 2 Here we are on the day 2 of Advent of SQL As I said in the previous day this is in SQLite so I won't be doing it in the playground. S Wed, 17 Dec 2025 00:00:00 UTC ## SQLog: Advent of SQL Day 2 Here we are on the day 2 of Advent of SQL As I said in the previous day this is in SQLite so I won't be doing it in the playground. So here is your SQLite playground :) ```sql SELECT 1; ``` From now on no setup straight to the problem! Let's download the .sql file for today's problem to see what data we are playing with. ```sql DROP TABLE IF EXISTS snowball_categories; DROP TABLE IF EXISTS snowball_inventory; CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); ``` Well, neat and clean! Straight pulling it in the sqlite shell. ```plaintext $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day2-inserts.sql sqlite> .schema CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); sqlite> ``` Worked right off the bat. Straight to problem now. If you are working in the playground, do add this full code with inserts in order to get a sense of what the data looks like, note that this below statements is not all data, just sharing to get you a taste of the problem. ```sql DROP TABLE IF EXISTS snowball_categories; DROP TABLE IF EXISTS snowball_inventory; CREATE TABLE snowball_categories ( id INT PRIMARY KEY, official_category TEXT ); CREATE TABLE snowball_inventory ( id INT PRIMARY KEY, batch_id TEXT, category_name TEXT, quantity INT, status TEXT ); INSERT INTO snowball_categories (id, official_category) VALUES (1, 'frost-flight deluxe'), (2, 'north ridge compact'), (3, 'glacier sphere (xl)'), (4, 'polar precision microball'), (5, 'everfrost training round'), (6, 'arctic blast premium'); INSERT INTO snowball_inventory (id, batch_id, category_name, quantity, status) VALUES (1, 'BATCH-35443-J', 'frost-flight deluxe', 19, NULL), (2, 'BATCH-59767-M', 'frost-flight deluxe', 41, 'incomplete'), (3, 'BATCH-44795-B', 'frost-flight deluxe', 21, 'ready'), (4, 'BATCH-23396-C', 'north ridge compact', 0, 'incomplete'), (5, 'BATCH-88907-A', 'frost-flight deluxe', -2, 'incomplete'), (6, 'BATCH-42662-D', 'frost-flight deluxe', 47, 'needs review'), (7, 'BATCH-37460-V', 'north ridge compact', 43, 'ready'), (8, 'BATCH-21395-S', 'frost-flight deluxe', -2, 'ready'), (9, 'BATCH-36100-E', 'frost-flight deluxe', 46, 'ready'), (10, 'BATCH-64987-H', 'frost-flight deluxe', 43, NULL), (11, 'BATCH-57576-Z', 'melty deluxe', -5, 'ready'), (12, 'BATCH-56025-U', 'snowball v2', 11, 'ready'), (13, 'BATCH-86556-W', 'snowball v2', 12, 'ready'), (14, 'BATCH-83385-N', 'frost-flight deluxe', 38, 'incomplete'), (15, 'BATCH-85156-M', 'prototype x-12', 28, 'incomplete'), (16, 'BATCH-82135-F', 'north ridge compact', 32, 'incomplete'), (17, 'BATCH-10074-T', 'frost-flight deluxe', 49, 'needs review'), (18, 'BATCH-22676-L', 'frost-flight deluxe', 16, 'incomplete'), (19, 'BATCH-31174-R', 'north ridge compact', 33, 'incomplete'), (20, 'BATCH-41385-B', 'frost-flight deluxe', 4, 'ready'), (21, 'BATCH-50404-L', 'frost-flight deluxe', -4, 'needs review'), (22, 'BATCH-92240-F', 'north ridge compact', 20, 'ready'), (23, 'BATCH-29198-J', 'beta test sphere', 0, 'incomplete'), (24, 'BATCH-64987-H', 'glacier sphere (xl)', 18, 'needs review'), (25, 'BATCH-80008-A', 'frost-flight deluxe', 3, 'incomplete'), (26, 'BATCH-88907-A', 'polar precision microball', 48, 'incomplete'), (27, 'BATCH-55830-J', 'north ridge compact', 0, 'needs review'), (28, 'BATCH-69470-A', 'frost-flight deluxe', -3, 'incomplete'), (29, 'BATCH-46211-R', 'frost-flight deluxe', -3, 'ready'), (30, 'BATCH-18675-G', 'glacier sphere (xl)', -1, 'ready'); ``` ## Problem Let's check the challenge of day 2 > Using the `snowball_inventory` and `snowball_categories` tables, write a query that returns valid snowball categories with the count of valid snowballs per category. Your final table should have the columns `official_category` and `total_usable_snowballs`. Sort the output from fewest to most `total_usable_snowballs`. So, we have two tables: 1. snowball categories 2. snowball inventory The snowball categories looks a quite a small table with just the category name and the id which is not really the data, the name is. ```sql SELECT * FROM snowball_categories; ``` Just 6 entries with some names of the categories, the column is official categories. ```sql sqlite> SELECT * FROM snowball_categories; +----+---------------------------+ | id | official_category | +----+---------------------------+ | 1 | frost-flight deluxe | | 2 | north ridge compact | | 3 | glacier sphere (xl) | | 4 | polar precision microball | | 5 | everfrost training round | | 6 | arctic blast premium | +----+---------------------------+ sqlite> ``` And the other table has oddles of data: ```sql SELECT * FROM snowball_inventory limit 10; ``` So, we have quite a few columns, 1. batch\_id 2. category\_name 3. quantity 4. status It has 200,000 rows, that's quite a lot. ```plaintext sqlite> SELECT count(*) FROM snowball_inventory; +----------+ | count(*) | +----------+ | 200000 | +----------+ sqlite> SELECT * FROM snowball_inventory limit 10; +----+---------------+---------------------+----------+--------------+ | id | batch_id | category_name | quantity | status | +----+---------------+---------------------+----------+--------------+ | 1 | BATCH-35443-J | frost-flight deluxe | 19 | | | 2 | BATCH-59767-M | frost-flight deluxe | 41 | incomplete | | 3 | BATCH-44795-B | frost-flight deluxe | 21 | ready | | 4 | BATCH-23396-C | north ridge compact | 0 | incomplete | | 5 | BATCH-88907-A | frost-flight deluxe | -2 | incomplete | | 6 | BATCH-42662-D | frost-flight deluxe | 47 | needs review | | 7 | BATCH-37460-V | north ridge compact | 43 | ready | | 8 | BATCH-21395-S | frost-flight deluxe | -2 | ready | | 9 | BATCH-36100-E | frost-flight deluxe | 46 | ready | | 10 | BATCH-64987-H | frost-flight deluxe | 43 | | +----+---------------+---------------------+----------+--------------+ sqlite> ``` What we have to do? > return valid snowball categories with the count of valid snowballs per category That's a mouthful. The problem is trying to get at counting the number of inventory items per category I think. Because there are quite less categories and tons of inventory records. Also the `category_name` in the `snowball_inventory` table is not trustworthy. > Santa hurried to the snowball storage board, but the situation only got stranger. Whole batches appeared twice. Some batches claimed they had negative snowballs (“a bookkeeping accident,” the elves muttered). Others had a quantity of zero but were still marked “Ready.” And many batches referenced categories that didn’t appear anywhere in the official Snowball Category Guide ## Naive Approach The first approach I see is scan through all the inventory records and check if the category is in the `snowball_category` table and its quantity is positive or more than 0, also I wonder if the status needs to be checked as `ready`. > “We need to know what we actually have left,” Santa said. “Not puddles. Not phantom batches. Real, usable, throw-ready snowballs.” **Real, throw-ready snowballs** Let's check how to do that with simple sub query. ```sql SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready'; ``` OK! What we did? Simply selected all the data from `snowball_inventory` table, in which the category name is matching either of the 6 categories in the `snowball_categories` table. Also the quantity is positive and not zero, also the status is set to be `ready`. This looks naive to me for using subquery because for each 200,000 records we will scan the `snowball_categories` . Ew! Wait its done! We need the count of each category! We need to group by the `category_name` ```sql SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; ``` ```sql sqlite> SELECT * FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; +-----+---------------+---------------------------+----------+--------+ | id | batch_id | category_name | quantity | status | +-----+---------------+---------------------------+----------+--------+ | 240 | BATCH-55793-L | arctic blast premium | 45 | ready | | 163 | BATCH-75333-O | everfrost training round | 23 | ready | | 3 | BATCH-44795-B | frost-flight deluxe | 21 | ready | | 39 | BATCH-22704-V | glacier sphere (xl) | 37 | ready | | 7 | BATCH-37460-V | north ridge compact | 43 | ready | | 125 | BATCH-81987-E | polar precision microball | 47 | ready | +-----+---------------+---------------------------+----------+--------+ ``` That sort of looks wired right? Why? Because what happens to the quantity? Is that summed? averaged, or what just happened to the batch\_id? No it takes one value out of the `200 000` rows for each category. That's not what we want right? We want this > return valid snowball categories with the count of valid snowballs per category So, we just want the category name and the count of those category. Basically count per category. ```sql SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; ``` Here we are only fetching the `category_name` and the `count` which will bundle up all the counts from the grouped of the category\_name. ```plaintext sqlite> SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | frost-flight deluxe | 952019 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | polar precision microball | 70773 | +---------------------------+------------------------+ sqlite> ``` Not done yet! We need to order by the count. > Sort the output from fewest to most `total_usable_snowballs`. ```sql SELECT category_name, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; ``` We can provide `ORDER BY total_usable_snowballs ASC` but ascending is default. I prefer keeping things default, you can be explicit and mention it as `ASC` to make it clear and readable. ```plaintext sqlite> SELECT category_name as official_category, sum(quantity) as total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ sqlite> ``` There it is! The solution to the day 2. But Santa, I am using subqueries. Is that fine? ## Joins? We can use joins here, since we just require the count of each category. Which JOIN though? LEFT, RIGHT, INNER? any really? You don't choose the join based on the problem, you define your output columns and then choose the type of join that would give you the result. If the thing that is to be searched from left to right and then its a left join, i.e. join everything from the left. and so on. Here, I need all the `official_category` column right, which is in the `snowball_categories` table. If I assume the `snowball_categories` is on the left, I can join everything for that row in the left to match all the `category_name` rows in the `snowball_inventory` table which would be in the right. Like so. ```sql SELECT snowball_categories.official_category as official_category, SUM(snowball_inventory.quantity) as total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` We JOIN on the condition of `snowball_categories.official_category = snowball_inventory.category_name` which is to say, the `category_name` in the `snowball_inventory` table should match the `official category` from the `snowball_categories` table. Also the other conditions like `quantity` should be more than `0` and the `status` should be `ready`. We will still need to group by and order by as the joins will basically be the filtering criteria in which to select or reject the bad categories. However to group by the category name and obtain the count of each category we need to group by the name of the category. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ ``` This is it! The same thing that we did for the subquery. Just the filtering part is different. How about INNER JOIN? `INNER JOIN` is different from `LEFT JOIN` as it will by default take only rows that match the condition, whereas `LEFT JOIN` will include records/rows from the `LEFT` table even if they are empty or `NULL` . Similar is `RIGHT JOIN`, it will include everything from the RIGHT table even if that is `NULL` so that all the records from the `RIGHT` table are in the result set. For INNER JOIN if we flip the tables end it won't matter, as it only relies on the condition and not on the order of where the tables are placed (left or right). ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` This is with INNER JOIN nothing changed here? did it? except the word `LEFT` is no longer there, the default join is `INNER JOIN` . In our case i don't think we have `NULL` values in the `snowball_categories` table or `snowball_inventory` table with the category name columns. So LEFT AND INNER JOIN won't be different in terms of the results. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ sqlite> ``` However if we decide to make the `snowball_inventory` table as the `Left` table then does it make a difference? Well we don't want that to be the left table if we are doing a `LEFT` JOIN as it might list all the wrong categories in the result list as well even if they are not in the `official_category` those will be 0 or NULL but that would make a mess. We instead can do a RIGHT JOIN. With the tables order as 1. snowball\_inventory 2. snowball\_categories And we will RIGHT JOIN that is select all the columns from the right to be in the result set even if the condition is not true. It will be NULL. But the case is not true for here, as there are only 6 non-NULL category names in the snowball\_categories table. ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ ``` However if we try to use `LEFT JOIN` with `snowball_inventory` as the `LEFT` table, we might get something wired. ```sql SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory LEFT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; ``` We have 6 and 1 more category as empty ? That category is for all the rows that didn't met the criteria of JOIN but still as asked to `LEFT JOIN` it will try to show the results for all the rows in the left table which is the `snowball_inventory` table. ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory LEFT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | | | 1692814 | +---------------------------+------------------------+ ``` ## Timer Phew! Let's see which one of them is the best one Let's see the timings: ```plaintext .timer on ``` ### SUBQUERY ```plaintext sqlite> SELECT category_name, SUM(quantity)AS total_usable_snowballs FROM snowball_inventory WHERE category_name IN (SELECT official_category FROM snowball_categories) AND quantity > 0 AND status == 'ready' GROUP BY category_name ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | category_name | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.077 user 0.073886 sys 0.002942 sqlite> ``` ### LEFT JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_categories LEFT JOIN snowball_inventory ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status == 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.182 user 0.171433 sys 0.009907 sqlite> ``` ### INNER JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.072 user 0.069623 sys 0.001929 sqlite> ``` ### RIGHT JOIN ```plaintext sqlite> SELECT snowball_categories.official_category AS official_category, SUM(snowball_inventory.quantity) AS total_usable_snowballs FROM snowball_inventory RIGHT JOIN snowball_categories ON snowball_categories.official_category = snowball_inventory.category_name AND snowball_inventory.quantity > 0 AND snowball_inventory.status = 'ready' GROUP BY official_category ORDER BY total_usable_snowballs; +---------------------------+------------------------+ | official_category | total_usable_snowballs | +---------------------------+------------------------+ | arctic blast premium | 11470 | | everfrost training round | 24248 | | polar precision microball | 70773 | | glacier sphere (xl) | 165158 | | north ridge compact | 594119 | | frost-flight deluxe | 952019 | +---------------------------+------------------------+ Run Time: real 0.151 user 0.149690 sys 0.001957 sqlite> ``` Time wise ranking: 1. INNER JOIN 2. SUBQUERY 3. RIGHT JOIN 4. LEFT JOIN That's quite quick, not much can be measured here. I am sure there are other ways to do this. But stopping here for today. Explored JOINs a little. See you for the day 3 sqlog https://www.meetgor.com/sqlog/advent-of-sql-2025-day-1 Learning SQLite: Advent of SQL Day 1 I am trying to learn SQLite, I want to understand that database. It's quite simple yet the whole world uses it for various Tue, 16 Dec 2025 00:00:00 UTC ## Learning SQLite: Advent of SQL Day 1 I am trying to learn SQLite, I want to understand that database. It's quite simple yet the whole world uses it for various kinds of things ranging from developers' toy database to spaceships. What a tiny engineering marvel! I am happy to see this happening: [Advent of SQL](https://databaseschool.com/series/advent-of-sql-videos-308) What a better time to learn more. I guess I want to start by exploring all the specificities of the INSERT statement in SQLite after exploring most of the things of the CREATE TABLE statement. But here I am jumping to this. Why? Because I want to solve something first before exploring other branches. Today I am going to try to solve the day 1 part. > **NOTE:** I would be using my local sqlite db for this or a playground on this for testing. I am not going to use the browser-based playground attached in the databaseschool.com app for a reason. I want to use SQLite. The database is some form of Postgres; I don't mind using it, but I want to do it in SQLite. I have a playground on my blog for SQLite, you can try it out here: ```sql SELECT 1; ``` It uses an embedded SQLite version (3.49.1) with sql-js as a wasm extension: [sql.js v1.13.0](https://github.com/sql-js/sql.js/releases/tag/v1.13.0) Back to the problem elves! ## Setup This is the first day, so advent calendar usually requires some setup or preparation for the rest of the days. Luckily it's optional for you if you are doing it in the playground of database school or in PostgreSQL Database. We have some .sql files as input for creation and insertions of tables and rows in the database. It's for constructing the schema (tables) and populating the rows that the problem requires us to do. The SQL looked something simpler like: ```sql DROP TABLE IF EXISTS wish_list CASCADE; CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (1, 'James A.', ' BLUEY SUPERMARKET PLAY SET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (2, 'Sade C.', 'lego star wars set '); ``` There are around 499,000 rows! However when I tried to read directly into a SQLite shell as: ``` .read day1-wish-list.sql ``` It got an error for parsing the DROP TABLE statement: ``` sqlite> .read day1-wish-list.sql Parse error near line 11: near "CASCADE": syntax error DROP TABLE IF EXISTS wish_list CASCADE; error here ---^ ``` Obviously it was designed for Postgres. It won't work in SQLite. SQLite is minimal. It might not have everything that PostgreSQL has, but PostgreSQL might have everything that SQLite has (maybe but not as is). So, we need to remove the CASCADE, which is an option to decide what to do with the related data rows when a relation is removed. In this case, it is cascading—deleting all the other related data points or records in the related tables. SQLite doesn't have options to modify the relations for the DROP TABLE statement. It has it for CREATE TABLE with the foreign key constraint. Now we need to remove it. It can't be in the DROP TABLE statement for SQLite database: ```sql DROP TABLE IF EXISTS wish_list; ``` Now, let's check by running the queries again: ``` .read day1-wish-list.sql ``` That works! ``` $ sqlite3 SQLite version 3.45.1 2024-01-30 16:01:20 Enter ".help" for usage hints. Connected to a transient in-memory database. Use ".open FILENAME" to reopen on a persistent database. sqlite> .read day1-wish-list_sqlite1.sql sqlite> .schema CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); ``` But this looks weird: ```sql CREATE TABLE wish_list ( id BIGSERIAL PRIMARY KEY, child_name TEXT, raw_wish TEXT ); ``` BIGSERIAL is not a datatype in SQLite. It might be in PostgreSQL. But does it matter? In SQLite, if the table is not STRICT, it doesn't matter what type the column is or even if it is NOT specified of any type. That's fine. INSERT some rows shall we? ```sql INSERT INTO wish_list (id, child_name, raw_wish) VALUES (1, 'James A.', ' BLUEY SUPERMARKET PLAY SET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (2, 'Sade C.', 'lego star wars set '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (3, 'Juan Q.', ' SCOOTER '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (4, 'Samir S.', ' LEGO STAR WARS SET '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (5, 'Priya E.', 'shaved ice machine '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (6, 'Henry L.', ' mini brands fill the fridge'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (7, 'Ayumi C.', 'VR HEADSET'); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (8, 'Juan Y.', 'BARBIE DREAMHOUSE '); INSERT INTO wish_list (id, child_name, raw_wish) VALUES (9, 'Priya O.', ' VR HEADSET '); ``` ```sql SELECT * FROM wish_list; ``` But I want to know what is the type of `wish_list.id`? ```sql SELECT distinct(typeof(id)) FROM wish_list; ``` Here's the output: ``` sqlite> SELECT distinct(typeof(id)) FROM wish_list; integer ``` It is integer, because of type affinity, I guess. Good work SQLite. Looks like the data is fine. Now to the problem. ## Problem [Link to the challenge](https://databaseschool.com/series/advent-of-sql-videos-309-text-challenge-using-the-and-get-started) **Challenge:** Using the wish_list table, count how many times each cleaned toy name appears, from most requested to least requested. Return the results in two columns: wish and count. Make sure the wish results have no extra leading or trailing spaces and are all lowercase. So, simply we need two columns: - **wish** (the text) - **count** (the number of times that wish is wished) ### Subtleties > Some children had typed extra spaces. Some wrote in ALL CAPS. Some had letters that danced between cases like playful snowflakes. > > I know there are some issues with spelling, the extra spaces, or the funny capitalization, but I just need to know what the children truly meant. So, we need to either use lower caps or upper caps the wish text and trim off the space. SCALAR FUNCTIONS!! ### Scalar Functions [SQLite Core Functions](https://sqlite.org/lang_corefunc.html) I read through the list of around ~70 of them, most of them are kind of the same with different parameters. The one that I found relevant are: - [LOWER](https://sqlite.org/lang_corefunc.html#lower) - [TRIM](https://sqlite.org/lang_corefunc.html#trim) That's it, right? Convert into LOWER (or UPPER) and TRIM off the spaces. ```sql SELECT LOWER(TRIM(raw_wish)) FROM wish_list; ``` Don't run just yet! ```sql SELECT LOWER(TRIM(raw_wish)) FROM wish_list LIMIT 100; ``` Looks good. Now to the next step. ### Grouping and Counting We need to count them i.e. to group by the wish. > GROUP BY: What group by does is that it condenses the rows of certain column into a singular column for instance if there are 10 entries for "lego star wars set" adding a group by wish will create a singular entry for that wish and we can then perform operations like sum, count, average and all of that on other rows ```sql SELECT LOWER(TRIM(raw_wish)) AS wish, count(*) AS count FROM wish_list GROUP BY wish; ``` Here we are grouping by wish because we don't want 10 entries of "lego star wars set" we just want one common entry to view the unique wishes. Also by using `COUNT(*)` we are counting the instances of each row. As I said, the multiple rows with the same wish are squished into a single row. So now we can use aggregate functions like count, sum, in our case we want to count how many instances of those particular wish are. ### Ordering Results Does that solve it? Mostly, just need the ORDER BY now. Because we also need to order the results: ```sql SELECT LOWER(TRIM(raw_wish)) AS wish, count(*) AS count FROM wish_list GROUP BY wish ORDER BY count DESC; ``` Perfect? Probably. > What ORDER BY does is that it just determines which way the row should be aligned based on what column and how i.e. the column name and either ASC(ending) or DESC(ending). Here we have ordered by count so that we can filter the most wished toy or least wished toy at the top and increasing or decreasing order of it. ### Results Now with the mode table: ``` sqlite> .mode table sqlite> SELECT LOWER(TRIM(raw_wish)) as wish, count(*) as count FROM wish_list group by wish order by count desc; +-----------------------------+-------+ | wish | count | +-----------------------------+-------+ | lego city f1 car | 32893 | | barbie dreamhouse | 32785 | | nerf blaster | 32746 | | lego star wars set | 32611 | | beyblade battle arena | 29564 | | magna-tiles pet playhouse | 29529 | | bluey supermarket play set | 26292 | | lego friends amusement park | 25982 | | pokemon trainer box | 25968 | | duplo building set | 23005 | | mini brands fill the fridge | 22965 | | electric toy train set | 22885 | | toniebox audio player | 19529 | | scooter | 19496 | | vr headset | 16468 | | squishmallows | 16304 | | shaved ice machine | 16263 | | drone for kids | 13151 | | coding robot | 13025 | | headphones | 13006 | | interactive robot dog | 9770 | | fidget spinner | 3590 | | yo-yo | 3565 | | slime kit | 3553 | | littlest pet shop playset | 3543 | | chatter telephone | 3527 | | fingerlings robot monkey | 3511 | | rubik's revolution | 3474 | +-----------------------------+-------+ sqlite> ``` --- Day 1 done, moving on to day two by helping those pesky elves tomorrow. I am amazed at how stupider problems humans create with those elves as target. Just kidding, humans are elves :) Happy Coding :) Happy Squealing sqlog